Need help with textricator?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

measuresforjustice
241 Stars 24 Forks GNU Affero General Public License v3.0 59 Commits 5 Opened issues

Description

Textricator is a tool to extract text from documents and generate structured data.

Services available

!
?

Need anything else?

Contributors list

Textricator

Textricator is a tool to extract text from documents and generate structured data.

If you have a bunch of PDFs with the same format (or one big, consistently formatted PDF) and you want to extract the data to CSV or JSON, Textricator can help! It can even work on OCR'ed documents!

Textricator is released under the GNU Affero General Public License Version 3.

Textricator is deployed to Maven Central with GAV

io.mfj:textricator
.

This application is actively used and developed by Measures for Justice. We welcome feedback, bug reports, and contributions. Create an issue, send a pull request, or email us at [email protected]. If you use Textricator, please let us know. Send us your mailing address and we will mail you a sticker.

io.mfj.textricator.Textricator
is the main entry point for library usage.

io.mfj.textricator.cli.TextricatorCli
is the command-line interface.

The CLI has three subcommands, to use the three main features of Textricator: * text - Extract text from the PDF and generate JSON. * table - Parse the text that is in columns and rows. See Table section. * form - Parse the text with a configured finite state machine. See Form section.

Quick Start

  • Install Java (version 8+)
    • Windows & Macos: Download from https://java.com and install.
    • Linux: Use your package manager.
  • Download the latest build of Textricator from https://repo1.maven.org/maven2/io/mfj/textricator/ - click on the directory for the latest version and download
    textricator-VERSION-bin.tgz
    (or
    textricator-VERSION-bin.zip
    for Windows).
  • Extract it.
  • Run a shell
    • Windows: run Windows Powershell (it should be in the start menu)
    • The following examples start with
      ./textricator
      . On Windows, use
      .\textricator.bat
      .
    • MacOS: Run Terminal (type "terminal" in Spotlight)
  • Show help
    • ./textricator --help
  • Download the example files to the textricator directory:
    • https://github.com/measuresforjustice/textricator/raw/master/src/test/resources/io/mfj/textricator/examples/school-employee-list.pdf
    • https://raw.githubusercontent.com/measuresforjustice/textricator/master/src/test/resources/io/mfj/textricator/examples/school-employee-list.yml
  • Extract raw text from a PDF to standard out
    • ./textricator text --input-format=pdf.pdfbox school-employee-list.pdf
  • Parse a PDF to CSV
    • ./textricator form --config=school-employee-list.yml school-employee-list.pdf school-employee-list.csv
      • This uses the configuration file
        school-employee-list.yml
        to parse
        school-employee-list.pdf
        . To parse your own PDF form, you will need to write your own configuration file. See the Form section for details. If your PDF has a tabular layout, see the Table section.

Table

In table mode, the data is grouped into columns based on the x-coordinate of the text.

Example

# All measurements are in points. 1 point = 1/72 of an inch.
# x-coordinates are from the left edge of the page.
# y-coordinates are from the top edge of the page.

Use the built-in pdfbox extractor

extractor: "pdf.pdfbox"

Ignore everything above 85pt from the top

top: 85

Ignore everything below 553pt from the top

bottom: 553

ignore to the right of 720pt from the left edge.

right: 720

There is no "left", because everything to the left of the first column is ignored.

pageConfig: 1: # On page 1, ignore everything above 100pt from the top. Overrides top:85 above. top: 100 # could also define bottom and right here. 123: # skip page 123 entirely. skip: true

If multiple text segments are withing 2pt vertically, consider them in the same row.

maxRowDistance: 2

Define the columns, based on the x-coordinate where the column starts:

cols: "bookingNum": 27 "name": 83 "class": 247 "sex": 359 "race": 389 "dob": 420 "bookDate": 475 "arrestingOfficer": 545 "releaseDate": 689

types:

Set bookingNum type so it can be used in filter

"bookingNum": type: "integer"

Do not include arrestingOffcier

"arrestingOfficer": include: "false"

"bookDate": label: "Booking Date"

Replace "M" with "Male", "F" with "Female"

"sex": replacements: - pattern: "M" replacement: "Male" - pattern: "F" replacement: "Female" - pattern: "^(.*)$" replacement: "Unknown: $1"

Omit if bookingNum is negative

filter: 'bookingNum > 0'

Form

In form mode, the data is parsed by Textricator using a finite-state machine (FSM), and the FSM and additional parsing and formatting parameters are configured with YAML, indicated by command line option

--config
.

Conditions

State transitions are selected by evaluating conditions. Conditions are expressions parsed by Expr.

Available variables:

  • ulx
    - x coordinate of the upper-left corner of the text box
  • uly
    - y coordinate of the upper-left corner of the text box
  • lrx
    - x coordinate of the lower-right corner of the text box
  • lry
    - y coordinate of the lower-right corner of the text box
  • text
    - the text
  • page
    - page number
  • page_prev
    - page number of the previous text
  • fontSize
    - font size
  • font
    - font name
  • color
    - text color
  • bgcolor
    - background color
  • width
    - width of the text box
  • height
    - height of the text box
  • ulx_rel
    - difference in
    ulx
    between the previous and current texts
  • uly_rel
    - difference in
    uly
    between the previous and current texts
  • lrx_rel
    - difference in
    lrx
    between the previous and current texts
  • lry_rel
    - difference in
    lry
    between the previous and current texts
  • added Variables

Example

This is an example for

src/test/resources/io/mfj/textricator/examples/school-employee-list.pdf
.
# Use the built-in pdfbox parser
extractor: "pdf.pdfbox"

All measurements are in points. 1 point = 1/72 of an inch.

x-coordinates are from the left edge of the page.

y-coordinates are from the top edge of the page.

header: # ignore anything less than this many points from the top, default and per-page default: 130 footer: # ignore anything less than this many points from the bottom, default and per-page default: 700

Text segments are generally parsed in order, top to bottom, left to right.

If two text segments have y-coordinates within this many points, consider them on the same line,

and process the one further left first, even if it is 0.4pt lower on the page.

maxRowDistance: 2

Define the output data record.

Since the main record type we're collecting information on is our employees,

we'll have that be the root type for our harvested information.

rootRecordType: employee recordTypes: employee: label: "employee" # Labels are used when nested recordTypes come into play, like this document. valueTypes: # Not sure what to name a valueType? Just make something up! - employee - name - hiredate - occupation - showinfo - bool1 - bool2 - bool3 - salary children: # In this example, there are multiple children nested under an employee, # so we'll treat it as a 'child' to the 'employee' recordType. - child child: label: "child" valueTypes: - child - grade

valueTypes: employee: # In the CSV, use "Employee ID" as the column header instead of "employee". label: "Employee ID" name: label: "Name" hiredate: label: "Hire Date" occupation: label: "Occupation" salary: label: "Salary" showinfo: label: "Important Info?" bool1: label: "Boolean 1" bool2: label: "Boolean 2" bool3: label: "Boolean 3" child: label: "Attending Child" grade: label: "Grade"

Now we define the finite-state machine

Let's name the state that our machine starts off with:

initialState: "INIT"

When each text segment is encountered, each transition for the current state is checked.

states: INIT: transitions: # The first bit of text we reach is 'ID-0001', so we'll try the only transition that should work here. - # If this condition matches (which it should) condition: employee # Curious about the condition? Sxroll further down to the conditions section of this YAML. # Then we'll switch to the 'employee' state! nextState: employee

employee: # ID number with the format 'ID-####' startRecord: true # When we enter this stage, we'll create a new "case" record. transitions: - # Now we move on to the name label. Once again, by varifying the condition and moving on after that. condition: namelabel nextState: namelabel

namelabel: include: false # The label isn't important information in and of itself, so we can just not include it in the data. transitions: - condition: name nextState: name

name: transitions: - # Sometimes a name will be in two segments, and we'll hit another 'name' text segment before anything else. # In that case, a state can transition to itself, compounding the information picked up in it. condition: name nextState: name - # Does the first condition not match the text? We move onto the next one. condition: hiredateLabel nextState: hiredateLabel

hiredateLabel: include: false transitions: - condition: hiredateLabel nextState: hiredateLabel - condition: hiredate nextState: hiredate

hiredate: transitions: - condition: occupationLabel nextState: occupationLabel

occupationLabel: include: false transitions: - condition: occupation nextState: occupation

occupation: transitions: - condition: occupation nextState: occupation - # This state and the next are an example of how you can choose, using conditions, what to include or not. # They share the same area of a document, but have qualities to them that can be distinguishable. # Does it meet 'showinfo' conditions? Then we go to the 'showinfo' state that includes it. condition: showinfo nextState: showinfo - # Doesn't meet 'showinfo'? Then check for 'notinfo' and exclude it. condition: notinfo nextState: notinfo showinfo: transitions: - condition: showinfo nextState: showinfo - condition: bool1 nextState: bool1 notinfo: include: false transitions: - condition: notinfo nextState: notinfo - condition: bool1 nextState: bool1

bool1: transitions: - condition: bool2 nextState: bool2 bool2: transitions: - condition: bool3 nextState: bool3 bool3: transitions: - condition: salaryLabel nextState: salaryLabel

salaryLabel: include: false transitions: - condition: salary nextState: salary

salary: transitions: - condition: childrenLabel nextState: childrenLabel - condition: employee nextState: employee - condition: end nextState: end

childrenLabel: include: false transitions: - condition: childrenLabel nextState: childrenLabel - condition: childLabel nextState: childLabel

childLabel: include: false transitions: - condition: child nextState: child

child: # Here we reach a datatype nested within another datatype. We can start records using this child datatype. # In the process, we'll be making multiple rows for the parent datatype, each one holding onto it's own child. startRecord: true transitions: - condition: child nextState: child - condition: gradeLabel nextState: gradeLabel - condition: childLabel nextState: childLabel

gradeLabel: include: false transitions: - # Normally, there would always been an instance of a grade appearing right after the label. # But in this document, we have one instance of that not happening under ID-0007's child. condition: grade nextState: grade - # So we just account for that possibility by adding a transition out of the label. condition: employee nextState: employee

grade: transitions: - condition: employee nextState: employee - condition: childLabel nextState: childLabel - # Reach the end of the usable info in a document, but there's still text left to go? # An easy fix is to just create a looping, not-included state to finish the document off. condition: end nextState: end

end: # We reached a point in the document where all the useful information is gone, but we still have text to go. include: false transitions: - # By using an always-true condition such as 'any', we can loop this state until the document has been completely gone through. condition: any nextState: end

Here we define the conditions:

conditions:

An example of comparing text with regex.

In this case, we're making sure that the text contains the characters 'ID-' followed by any amount of numbers.

employee: 'text =~ /ID-(\d)*/'

You can match based on the x- and y- coordinates of the upper left and lower right corners of the rectangle

containing the text. ulx = Upper-Left X-coordinate. lry = Lower-Right Y-coordinate. Also uly and lrx.

You can define the lower and upper limit for each, inclusive.

namelabel: '70 < ulx < 80 and font = "BCDFEE+Calibri-Bold"'

You can also match based on the type of font used, including if it was bolded or italicized.

name: '112 < ulx < 200 and font = "BCDEEE+Calibri"'

hiredateLabel: '230 < ulx < 270 and font = "BCDFEE+Calibri-Bold"'

hiredate: '280 < ulx < 290 and font = "BCDEEE+Calibri"'

occupationLabel: '391 < ulx < 393 and font = "BCDFEE+Calibri-Bold"'

occupation: '394 < ulx < 700 and font = "BCDEEE+Calibri"'

showinfo: 'font = "BCDJEE+Georgia"'

notinfo: 'font = "BCDEEE+Calibri"'

bool1: 'font = "BCDIEE+Cambria"'

bool2: 'font = "BCDIEE+Cambria"'

bool3: 'font = "BCDIEE+Cambria"'

salaryLabel: '391 < ulx < 393 and font = "BCDFEE+Calibri-Bold"'

salary: '394 < ulx < 700 and font = "BCDEEE+Calibri"'

childrenLabel: '70 < ulx < 140 and font = "BCDFEE+Calibri-Bold" and text =~ /(Attending)|(Children:)/'

childLabel: '230 < ulx < 240 and font = "BCDFEE+Calibri-Bold"'

child: '230 < ulx < 380 and font = "BCDEEE+Calibri"'

gradeLabel: '391 < ulx < 393 and font = "BCDFEE+Calibri-Bold"'

grade: '394 < ulx < 700 and font = "BCDEEE+Calibri"'

You can also match based on the size of the font and on specific text.

end: 'fontSize = 16.0 and text = "TOTAL:"'

Need a condition that is always true? "1=1" does that for you.

any: "1 = 1"

Advanced functionality

Regex matching

You can use regular expression matching instead of exact string matching in conditions:

conditions:
  caseTypeLabel: 'text =~ /Case Type:?/' # maybe sometimes they forgot the ":"

Font matching

You can match font and font size:

conditions:
  helvetica8: 'font =~ /.*Helvetica.*/ and fontSize = 8'

Page number matching

You can match the page number

conditions:
  # Match the textbox that starts at 100pt,120pt on page 9.
  specificTextbox: 'page = 9 and ulx = 100 and uly = 120'

Requiring intermediate state

There may be cases where you want to transition from state "B" to state "A" and start a new record, but ONLY if you were in state "C" since last starting a new record:

states:
  A:
    startRecord: true
    startRecordRequiredState: C
    # ...

Combining states

You may have 2 different states that you want to combine into one column in the output:

recordTypes:
  record:
    label: "Record"
    valueTypes:
      - A
      # no A2
      - B

states: A: # ... A2: valueTypes: [ A ] # combine this with state "A". # ...

Joining text segments in a state

When there are multiple text segments in the same state, by default they are concatenated with a space in between. Any string, including an empty string, can be used as the separator:

valueTypes:
  recordId:
    # concatenate without spaces between
    label: "Record ID"
    separator: ""
  description:
    # put newlines between text segments
    label: "Description"
    separator: "
"

Exclude types from output

A value type (

dataRecordMember
) can be excluded from the output. This is useful if the type is repeated on the PDF and needed as a data type to set new records, but should not be in the output.
valueTypes:
  repeatedId:
    include: false

Relative offsets

Conditions can evaluate coordinates relative to the coordinates of the previous text. This is useful for matching only something on the next line:

maxRowDistance: 2
conditions:
  # These should generally be the same (absolute value) as maxRowDistance
  descriptionSameLine: '-2 <= uly_rel <= 2'
  descriptionOneLineDown: '12 <= uly_rel <= 16' # for 12pt, single-spaced font.

Control which page number in output

If a complex data record spans multiple pages, which page number is used for the output can be controlled. Each type (

dataRecordType
) has a page priority (default: 0). The page number for the record comes from the type with the highest priority.
recordTypes:
  agency:
    label: "Agency"
    valueTypes:
      - agencyName
    children:
      - case
    # Agency may span hundreds of pages
    # default page priority of 0
  case:
    label: "Case"
    valueTypes:
      - caseNumber
      - name
    children:
      - charge
    # Case record may span multiple pages.
    # Higher priority than agency but lower than charges,
    so case is used if there are no charges.
    pagePriority: 1
  charge:
    label: "Charge"
    valueTypes:
      - chargeNumber
    # Highest priority. Use charge's page number.
    pagePriority: 2

Starting new records for each value

TODO

Regex replacement of values

Sometimes the value contains a label, or other text you want to remove. In a

dataRecordMember
, add a
replacement
, which has a
pattern
, which is a regular expression with capturing groups, and
replacement
, which is a replacement string with group references. See
java.util.regex.Pattern
and
java.util.regex.Matcher
for details.
# Replace "Bond Agency: Fred's Bonds" with "Fred's Bonds"
valueTypes:
  bondagency:
    label: "Bond Agency"
    replacements:
      -
        pattern: "Bond Agency:\ *(.*)"
        replacement: "$1"

Variables

On entering a state, a variable can be set.

In

State
, add a
setVariable
, which has
name
of the variable and the
value
to set.

If the

value
starts with
{
and ends with
}
, the content can be any built-in variable or previously set
variable
- the same things usable as variables for conditions.

For example: a condition checks that has the same background color as the last caseNo:

states:
  caseNo:
    setVariables:
      -
        name: "lastCaseBgColor"
        value: "{bgcolor}"

conditions: sameBgColor: > 145 <= ulx and fontSize = 8 and bgcolor = lastCaseBgColor

Splitting one PDF field into multiple CSV columns

One field in the PDF can be broken up into multiple columns in the CSV file, based on different regular expressions and replacements:

# Take a field that contains lastName,firstName and split it into 2 fields - lastName and firstName
recordTypes:
  inmateName:
    label: "inmateName"
    valueTypes:
      - lastName
      - firstName

valueTypes: lastName: label: "Last Name" replacements: - # lastName column will contain what's before comma pattern: "(.),." replacement: "$1" firstName: label: "First Name" replacements: - # firstName column will contain what's after comma pattern: ".,(.)" replacement: "$1"

states: name: # when FSM hits state "name", split data into lastName and firstName valueTypes: - lastName - firstName

Hyperlinks

If the text is a hyperlink, the URL can be used intead of the text by setting

valueTypes.attribute
to
url
.

This is supported only by the

itext5
and
itext7
parsers.
recordTypes:
  company:
    label: "Company"
    valueTypes:
      - name
      - website

valueTypes: name: label: "Company Name" website: label: "Website" attribute: url # use the link URL instead of the text.

states: name: # Put the value in both "name" and "website" to get both the # text and the url into the output. valueTypes: - name - website

Filtering records

Records can be filtered before output by setting a filter on the

dataRecordType
. Root records with a non-matching filter will not be output. Child records with a non-matching filter will be removed from their parent - the root will still be output.

Filters are expressions parsed by Expr

The variables are the fields in the record. The default type for variables is

STRING
. The type can be set by setting
type
in the
dataRecordMember
to
string
,
integer
, or
double
.
recordTypes:
  case:
    label: "Case"
    # Only include cases in 2009-2013
    filter: "2009 <= year <= 2013"
    valueTypes:
      - year
      - name
    children:
      - charge
  charge:
    label: "Charge"
    # omit charges of type "dummy"
    filter: >
      not( type = "dummy" )
    valueTypes:
      - code
      - type

valueTypes: year: type: integer

Filtering happens after replacements and before member include checking. This allows splitting a field (e.g.: extracting a year from a date) to use in the filter, and not including the split field in the output.

Extractors

Extractors extract text (instances of

io.mfj.textricator.text.Text
) from a source. There are four included extractors:
  • pdf.pdfbox - Extract text from PDF files using Apache PDFBox.
  • pdf.itext5 - Extract text from PDF files using iText 5.
  • pdf.itext7 - Extract text from PDF files using iText 7.
  • json - Parse text from the JSON format generated by the "text" subcommand of the CLI and by
    io.mfj.textricator.Textricator.parseText()
    .

Other extractors modules may be loaded, which may support different source types, capture different information or split the text up differently. An extractor module is loaded by adding a properties file

io/mfj/textricator/extractor/textExtractor.properties
to the classpath with a single property - the key is the extractor name and the value is the fully-qualified class name of an implementation of
io.mfj.textricator.extractor.TextExtractorFactory
. Typically an extractor module will be distributed as a JAR that includes
textExtractor.properties
.

Indicate which extractor to use by setting

extractor
in the yaml configuration, or overriding it by passing the extractor name to the
inputFormat
/
--input-format
option. If the extractor is not specified and the input is PDF,
pdf.itext5
is used.

Versioning

Textricator's version is of the format

major.minor.build
.

The major number is incremented for breaking changes or major new features.

The minor number is incremented for minor new features.

The build number is generated by Measures for Justice's private CI tool. It is incremented for each build, regardless of the major and minor numbers (it does not reset to zero when minor or major numbers are increased).

Acknowledgements

Much credit is due to some people who do not show up in the commit history:

  • Joe Hale, for the original idea and prototype
  • John Castaneda and Abbie Miehle who, as the first end users, provided excellent bug reports, improvements, documentation, and examples.

mascot

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.