User:Pedro Sá Couto/TW/JSTOR De-watermarking

From XPUB & Lens-Based wiki

STEPS

De-watermarking is separated into 4 steps:

1. Bursting the PDF into png
2. Overlaying the cover
3. Overlaying the pages
4. OCR again

RESULTS IN EACH STEP

0. Starting with a Paper from JSTOR
File:42938075.pdf

1. Bursting the PDF into PNGs

PDF is seperated into pages

2. Overlaying the cover

The cover is overlayed and dewatermarked

3. Overlaying the pages

The pages are overlayed and dewatermarked

4. OCR again

You have a De-watermarked, searchable PDF

File:42938075 dewater.pdf

FLOW

JSTOR.SH

To activate the stream I use ./jstor.sh

It is on a bash for loop. In this way I can easily control each 5 outputs.

for i in {1..5}
do
  cd /Users/PSC/Desktop/JSTOR/jstorpaper/
  for name in *; do mv "$name" "${name// /_}"; done
  cd `ls -td -- /Users/PSC/Desktop/JSTOR/jstorpaper/* | head -n 1`
  var2=`ls -td -- /Users/PSC/Desktop/JSTOR/jstorpaper/* | head -n 1`
  for name in *; do mv "$name" "${name// /_}"; done
  cp $var2/*.pdf /Users/PSC/Desktop/JSTOR/overlay
  cd /Users/PSC/Desktop/JSTOR/overlay
  mv /Users/PSC/Desktop/JSTOR/overlay/*.pdf target.pdf
  mkdir -p split
  python3 burstpdf.py
  python3 overlaylogo_cover.py
  python3 overlaylogo_page.py
  rm target.pdf
  convert "split/*.{png,jpeg,pdf}" -quality 100 name.pdf
  var1=$var2/*.pdf
  mv name.pdf $var1
  rm -r split
  mv $var2 /Users/PSC/Desktop/JSTOR/ready
  cd /Users/PSC/Desktop/JSTOR/ready
  ocrmypdf `ls -td -- /Users/PSC/Desktop/JSTOR/ready/*/*.pdf | head -n 1` `ls -td -- /Users/PSC/Desktop/JSTOR/ready/*/*.pdf | head -n 1`
done


1. Bursting the PDF into png

#Based in the code in https://iq.opengenus.org/pdf_to_image_in_python/

import pdf2image
from PIL import Image
import time

#DECLARE CONSTANTS
PDF_PATH = "target.pdf"
DPI = 200
FIRST_PAGE = None
LAST_PAGE = None
FORMAT = 'png'
THREAD_COUNT = 1
USERPWD = None
USE_CROPBOX = False
STRICT = False

def pdftopil():
    #This method reads a pdf and converts it into a sequence of images
    #PDF_PATH sets the path to the PDF file
    #dpi parameter assists in adjusting the resolution of the image
    #first_page parameter allows you to set a first page to be processed by pdftoppm
    #last_page parameter allows you to set a last page to be processed by pdftoppm
    #fmt parameter allows to set the format of pdftoppm conversion (PpmImageFile, TIFF)
    #thread_count parameter allows you to set how many thread will be used for conversion.
    #userpw parameter allows you to set a password to unlock the converted PDF
    #use_cropbox parameter allows you to use the crop box instead of the media box when converting
    #strict parameter allows you to catch pdftoppm syntax error with a custom type PDFSyntaxError

    start_time = time.time()
    pil_images = pdf2image.convert_from_path(PDF_PATH, dpi=DPI, first_page=FIRST_PAGE, last_page=LAST_PAGE, fmt=FORMAT, thread_count=THREAD_COUNT, userpw=USERPWD, use_cropbox=USE_CROPBOX, strict=STRICT)
    print ("Time taken : " + str(time.time() - start_time))
    return pil_images

def save_images(pil_images):
    d = 1
    for image in pil_images:
        image.save(("split/page%d"%d) + ".png")
        d += 1

if __name__ == "__main__":
    pil_images = pdftopil()
    save_images(pil_images)


2. Overlaying the cover

from PIL import Image

background = Image.open("split/page1.png")

#rescaling the logo
basewidth = (background.size[0])
baseheight = (background.size[1])
finalcover = Image.open("cover.png")
wpercent = (basewidth/float(finalcover.size[0]))
hsize = int((float(finalcover.size[1])*float(wpercent)))
finalcover = finalcover.resize((basewidth,hsize), Image.ANTIALIAS)
finalcover.save("cover_rescale.png")

foreground = Image.open("cover_rescale.png")
foregroundheight = (foreground.size[1])

background.paste(foreground, (0, (baseheight-foregroundheight)), foreground.convert('RGBA'))
background.save("split/page1.png")


3. Overlaying the pages

This happens through ./jstor.sh

from PIL import Image

base = Image.open("split/page2.png")

#rescaling the logo
basewidth = (base.size[0])
baseheight = (base.size[1])
finalpage = Image.open("pages.png")
wpercent = (basewidth/float(finalpage.size[0]))
hsize = int((float(finalpage.size[1])*float(wpercent)))
finalpage = finalpage.resize((basewidth,hsize), Image.ANTIALIAS)
finalpage.save("page_rescale.png")

foreground = Image.open("page_rescale.png")
foregroundheight = (foreground.size[1])

i = 2

while True:
    try:
        background = Image.open("split/page%i.png"%i)

        background.paste(foreground, (0, (baseheight-foregroundheight)), foreground.convert('RGBA'))
        background.save("split/page%i.png"%i)

        i+=1

    except:
        print("DID MY JOB!")
        break


4. OCR again

This happens through ./jstor.sh

ocrmypdf `ls -td -- /Users/PSC/Desktop/JSTOR/ready/*/*.pdf | head -n 1` `ls -td -- /Users/PSC/Desktop/JSTOR/ready/*/*.pdf | head -n 1`