User:Andre Castro/2.1/research-experiments-log
Experiment's Log
One experiment per day
03.10.2012
Sharing my digital library
Steps
- put my calibre ebook library on machine online for 24h
- start calibre content server
$ calibre-server
which point to my Calibre-library, and uses port machine_ip:8080
- check library remotely
Problem!! Calibre library is only accessible on the local LAN where I am at. http://www.mobileread.com/forums/showthread.php?t=160387
- in order to access the library remotely I have to have calibre installed on a server.
- calibre server on a LAN becomes a bit redundant, perhaps is handy for grabbing the books to ereaders and exchange them with people that are near one (on the same LAN), but one cannot say is yet a strong strategy for sharing books with someone in another part of the world.
04.10.2012
opening and modifying and epub
Too much talk about ebooks, witout actually looking at the insides of one. That's what I will do, will and create a new one with a section of the original.
- choose a book: Japanese Fairy Tales by Yei Theodora Ozaki
dissecting the epub
unzip the epub: there are 2 directories;
A) 4018/ - content dir
A.1) html ,css - html content and style
A.2) content.opf - Open Packaging Format metadata file (can be called anything, but content.opf is the convention) - specifies the location of all the contents in the book + the metadata (in xml)< br/>
A.2.1) metadata: required terms:title and identifier(the identifier must be a unique value, although it's up to the digital book creator to define that unique value)
A.2.2) manifest: all the content files par of the book
A.2.3) spine: indicates the order files they appear in the ebook - but not extraneous (like begging and end)
A.2.4) guide: (not required) explains what each section means
A.3) toc.ncx - The NCX defines the table of contents, but also metadata (overlaps w/ content.opf)
A.3.1) metadata- requires:
- uid: unique ID for the digital book. Should match the dc:identifier in the OPF file.
- depth: the level of the hierarchy in the table of contents
- totalPageCount and maxPageNumber: only to paper books and can be left at 0.
A.3.2) navMap: contains the navPoints
A.3.2.1) navPoint:
- playOrder - reading order. (same as itemref elements in the OPF spine).
- navLabel/text describes the title of this book section, a chapter title or number
- content src attribute points to content file. (a file declared in the OPF manifest). (can also point to anchors within XHTML eg: content.html#footnote1.)
A.4)and cover;
B) META-INF/container.xml (pointing to content.opf) - EPUB reading systems will look for this file first, as it points to the location of the metadata for the digital book.
B.2) META-INF can contain file such as digital signatures, encryption, and DRM
C) ./mimetype - file containing 'application/epub+zip'
packaging the epub into an epub+zip file
- create the new ZIP archive and add the mimetype file (no compression)
$ zip -0Xq my-book.epub mimetype
- add the remaining items
$ zip -Xr9Dq my-book.epub *
-X and -D minimize extraneous information in the .zip file; -r adds files from dirs 9? q?
Conclusion
- in the epub the text overlaps heavily, maybe because I didn't add a font size to the css; (text-height:200%; solved it)
- however in a kindle looks fine. Why?because kindle is imposing its font on the text - 'If you are publishing to Kindle it forces your font into it's own custom font so it doesn't matter'. This show what close the device is.
- even tough the process of creating an epub and its structure are straight-forward, writing the .opf and .ncx without any automated process its a pain (but important to understand the epub's structure) and may easily lead to errors.
- also it seems not the easiest format to experiment with. The need to declare all the files used and the metadata as well as packaging, will make think twice before trying something out.
resources
http://www.ibm.com/developerworks/xml/tutorials/x-epubtut/index.html
05.10.2012
Began a ebook - Spam Life - A compilation of folk stories and fary-tales from the year 2012
Original Message --------
Subject: Contribute with spam emails
Date: Fri, 05 Oct 2012 18:36:49 +0200
From: andre castro <andrecastro83@gmail.com>
To: undisclosed-recipients:;
Hia,
I am emailing to ask you for a favor.
Do you happen to have received sometime recently spam email, in a quite
personal tone and addressing you? Do think they might be somewhere in
you inbox or spam?
If you do so, or receive one in the next few day, I'd like to ask you,
you to forward them to me as I am putting together a compilation of
those particular spam emails. They can be in any language
Thank you
Best
a
The result won't take long
09.10.12
I made a few attempts to work with Sigil to help me work in Spam book project, however its far from ideal. I adds excessive markup content by itself, and makes the whole business quite messy, specially when working with large amount of text, when automated processes are very handy.
What I will try to do today is divide the creation of an epub in 4 stages.
- Having all the content in plain text with mediawiki notation - mostly headings, bold, and italics
- use a parse to generate an html from the plain text file
- Use calibre to convert the html to epub.
- Question: What is the fundamental difference between and ebook and a website? Can't a website, mostly made of text and re-flowable , be a ebook? With perhaps even more potential and easier to experiment with?
10.10.12
Character Encoding of a image-pdf.
I wanted to get hold of Licklider Libraries of Future. The only online copy I found was a pdf constituted of scanned images. Thought this could be a good opportunity to get some hands on character recognition process.
NOTE: PDF MUST BE CONVERTED TO IMAGE AT 300PDI!!!
Steps
- extract separate images out of a pdf
- convert output images files to a TIFF (single-bit uncompressed) so the tesseract (ocr software) can read them.
- using imagemaick
convert -monochrome -density 600 source.pdf page.tif
-density refers to resolution of the scanned image (the higher the better) 600pdi
- using gimp (as documented in http://alexsleat.co.uk/2010/04/12/howto-simple-tesseract-usage-guide-ocr/ )
- converted the 2 images into a text file using tesseract
tesseract image.tif text
Outputs result in .txt format
Results
- gimp tiff: resulting text and image
but rather that publication has been extended far beyond our present ability to make real use of the record. The summa- tion of human experience is being expanded at a prodigious rate, and the means we use for threading through the con- sequent maze to the momentarily important item is the same as was used in the days of square-rigged ships.*
- imagemagick tiff: resulting text and image
§;;iuEy ix’: af ihszfé aiaximiai gzimi *»;<~°2:::ii?s,:s§;§-°‘ iéf §§§‘:‘T.°:f:3é‘;‘;i%E3§-°-£§3:3;}I imafiriégis, bait m1:E:::::* that Apwiiaaiim has bi?-iii} §:E;}i§:€i3fl§€3-ii fa: b€::«,.«~’§;3m?§ mu: p=:°‘@$e:m sgibifiiiy Ea mam ma} we Sf that rzaaazaig ’E’ha samzmaw timz Qf Emmazfa m:§§:r§V€.%:1a$ £3 baffsizzzg $.I><Z§&§‘fid€€§ at :5: pradigiaaza mm, and tha;: X?JE€;?3£:iE.§S ass: far fifamadizfag ifiarmzgfiz ‘mg: mam wqumii mam‘: ix} th.s::‘r 3:m‘m'2e:,*:12.€z21°§§:; impamgazzt itazm is that: Sams :13 wzag Lzsmi in am: days mi Szzgzarawrigggad giaipgfi‘
Conclusion
In order to be able to have the whole book translated into text I will have to fine-tune imagemagick conversion to TIFF, so that tesseract does a better job, in character recognition.
Just as to see how would tesseract would react to a scanned TIFF, I scanned a printed text on a table scanner at 600pdi directly to a TIFF file. The result was very good
Here is the both scan and text
Introduction In the preface to A Contribution to the Critique of Political Economy, Marx argues that, ‘at a certain stage of development, the material productive forces of society come in conflict with the existing relations of production’? What is possible in the information age is in direct con- flict with what is permissible. Publishers, film producers and the telecommunication indus- try conspire with lawmakers to bottle up and sabotage free networks, to forbid information from circulating outside of their control. The corporations in the recording industry attempt to forcibly maintain their position as mediators between artists and fans, as fans and artists merge closer together and explore new ways of interacting.
11.10.12 - Streamline - character Encoding of a image pdf
Yesterday I was making a crucial mistake, and that's why I have been getting so many encoding errors. I was importing my pdf to tifs at 100dpi, creating a very low-res document, therefore hard for the ocr software to recognize the characters.
So now I have to:
- seperate the entire pdf into seperate pdf files
pdftk Licklider.pdf burst
- remove irrelevant pages (from beginning and end)
- convert each individual pdf to gray(? does it need the gray) 300dpi tifs
convert -units pixelsperinch -density 300x300 -colorspace Gray -depth 8 Licklider_10.pdf Licklider_10-gray-8.tif
for i in pg*.pdf;do convert -units pixelsperinch -density 300x300 -colorspace Gray -depth 8 $i "`basename $i .pdf`.tif";done // this will take a while now
- convert tif to monochrome
convert Licklider_10-gray-8.tif +dither -monochrome -normalize Licklider_10-gray-8-MONO.tif
for i in pg*.tif;do convert $i +dither -monochrome -normalize "`basename $i .tif`-m.tif";done
- ocr the image using tessaract
tesseract Licklider_10-gray-8-MONO.tif Licklider_10-gray-MONO
for i in *-m.tif;do tesseract $i "`basename $i .tif`"; done
- cat all txts into one
touch full_text; cat *.txt > full_text
Foreword THIS REPORT of research on concepts and problems of “Libraries of the Future†records the result of a two-year inquiry into the applicability of some of the newer tech- niques for handling information to what goes at present by the name of library work — i.e., the operations con- nected with assembling information in recorded form and of organizing and making it available for use. Mankind has been complaining about the quantity of reading matter and the scarcity of time for reading it at least since the days of Leviticus, and in our own day these complaints have become increasingly numerous and shrill. But as Vannevar Bush pointed out in the article that may be said to have opened the current campaign on the “information problem,†The diï¬iculty seems to be, not so much that we publish un- V
Conclusion
Although, after a bit of tweaking to get the book pages images processed for scaning, the ocr worked well, with only a few glitchs in the text, that need to be emended. A nice way to do this proofreading would be to edit the text while reading it on an ereader.
Besides scanning errors the correction there a few things that need to be done to the .txt before translating it into other formats
Preparing the text resulting from ocr
This process will dependent on the book we are dealing with, it will be hard to generalize.
- line breaks:
They are present throughout the book (including hypenated line breakes) and need to be remove.
import re, os
os.chdir(".")
for files in os.listdir("."):
if files.endswith("-m.txt"):
print files
text_file = open(files, 'r')
text = text_file.read()
# -\n : hyphen followed by new line
p_hyp_line_br = re.compile('-\n')
text = re.sub(p_hyp_line_br, '', text)
# just line breakes
p_line_br = re.compile('\n')
text = re.sub(p_line_br,' ', text)
sub_name_pre = files.split('.')
sub_name = sub_name_pre[0] + '_sub.txt'
sub_file= open(sub_name, 'w')
sub_file.write(text)
sub_file.close()
- page numbers:
before they were at the bottom, now with the substitution I did they became 'glued' to the text. I will leave them for now since they are also no very intrusive.
- Figures:
this depends on each book, if it has pictures and tehy are indicated
Libraries of the Future has images idicated as Fig
so I can run
and will get a list of the files where the pictures are. I will need to go back to pdf to get the pictures and edit the epub replacing the g\ ibrish orc replace them with, with the actual pictures
- remove page numbers - maybe it is not such a good idea, since its is good to keep the references and allows to refer back to the book
- remove the chapter headings (except their 1st occurrence)
- remove line breaks (including hyphenation) calibre took care of them
Calibre did a great job in converting the .txt file into a epub, even generating a table of contents. The only think that seems to be need is to remove the chapter headings, it shouldn't be hard!!
Reading spam
The email asking for spam email and the replies to it got me thinking of what happens to spam email when put under a different context than their original one, perhaps a more bookish form. Do we read them with different eye and pay attention to details that we have previously despised and begin to find some value to them?
I at least do, and find quite compelling to look as these considered minor for of (written) expression, but where also a lot of creativity, humor and invention appears to be put in place, often not for the most correct reasons.
An ongoing book
Lets say I want to go forward with the idea, mentioned on the previous post, of recontextualizing spam under a book format. Could quite some fun to place a book only consisting of spam on Kindle shop or Lulu, and see if anyone actually buys it.
Not despising the fact that I would be learning quite a bit from putting the book 'out there', it seems rather premature, and not exploiting that much the potential this material might have.
This can be a good source material to work on a web-book hybrid. The project could be materialized under the form of a website where the book-website could be in constant transformation, but also 'snapshots' from it could be take could be taken at any moment.
ideas for the project
contributions
- content contributions from anyone: new spam emails could be submitted and included into the project's contents
- notes on the text: anyone could make notes on specific parts of the text, which would become part of the project's notes' content. Eg: user can realize part of a spam email quotes the bible and note that down
- search on the select portions text: using google search, wikipedia, wiktionary.
snapshots
- at any moment the project content could be exported into an epub or pdf.
An ongoing book: questions
How is it different from a project like Booki? (have to get to know Bookie better for a proper answer). What it seems to me is that the technical structure is there, however both contents and forms present seem not that interesting. Perhaps the process through one contributes still has too much of the wait of 'writing a book' and therefore there isn't much movement or a feeling of a community around it.
Wikipedia seems to take that weight away from contributors. They are forced to write entire articles, but only little pieces of information, like pieces of a puzzle. Perhaps that's why it is so successful in hiring contributors.
Wikipedia articles are also a space for frequent debate.
Also Wikipedia now-a-day allow for articles to be exported as pdf or assembled into a book that one can buy from pediapress.
The difference between the project I am suggesting and these two examples doesn't obviously lie in the technical possibilities, but rather on content and form.
It tries not to be a functional service for writing books or constructing encyclopedic knowledge. It rather, it will try to look at the potential and specificities of marginal material. And the form it will try to call for a renovated attention, but also reflect the activity, such as notes and discussions around each item.
[See discussion of M.Fuller w/ Sean Dockray (aaaaaaarg), they refer the importance of community in wikipedia at some point;
Andrejevic's 'The work of Being Watched' - audiences work ]