16.
Creating Rules-Based Pipeline for Holocaust Documents

Dr. W.J.B. Mattingly
Smithsonian Data Science Lab and United States Holocaust Memorial Museum
January 2021

16.1. Key Concepts in this Notebook

  1. How to add pipes to a spaCy model

  2. How to Consider Rules

  3. How to Implement those Rules

16.2. Introduction

In this notebook, we will walk through some of the heuristic pipes I developed or am developing for my Holocaust NER spaCy pipeline. The purpose of these heuristic pipes is not to catch all potential entities, but to return with a high degree of confidence only true positives. We accept that the heuristics won’t catch everything because the final item in this long pipeline is a machine learning NER model that will generalize, or make predictions, on the unseen data. By structuring many heuristics in the pipeline, we can radically reduce the chances of our ML model making a wrong prediction because the all known true positives will have already been annotated.

I should also note that the code in my pipes is repetitious by design. As this is a textbook, it would be difficult for the reader to consistently look at the top of the notebook to identify the variable set 20 cells earlier. For this reason, I opt to recreate the variables later in the notebook. While this is bad practice, it allows the reader to understand better what is happening at any given moment in the notebook.

16.3. Creating a Blank spaCy Model

The first thing we need to do is import all of the different components from spaCy and other libraries that we will need. I will explain these as we go forward.

import spacy
from spacy.util import filter_spans
from spacy.tokens import Span
from spacy.language import Language
import re
import pandas as pd
INFO:tensorflow:Enabling eager execution
INFO:tensorflow:Enabling v2 tensorshape
INFO:tensorflow:Enabling resource variables
INFO:tensorflow:Enabling tensor equality
INFO:tensorflow:Enabling control flow v2

We will be using Pandas in this notebook to run data checks in a CSV file originally produced by the Holocaust Geographies Collaborative, headed by Anne Knowles, Tim Cole, Alberto Giordano, Paul Jaskot, and Anika Walke. We will be importing RegEx because a lot of our heuristics will rely on capturing multi-word tokens.

Now that we have imported everything, let’s create a blank English pipeline in spaCy. As we work through this notebook, we will add pipes to it.

nlp = spacy.blank("en")

16.4. Creating a Pipe for Finding Ships

The very first pipe I want to create is a pipe that can find ships, specifically transport ships. In order to achieve this objective, this pipe does several things at once. First, it leverages RegEx to find known patterns that are always ships, e.g. multi-word tokens that may or may not begin with “S.S.”, “SS”, or “The”, followed by a list of known ships. I have opted to not use a spaCy EntityRuler here primarily because I want to expand this pipe in the future and it allows me to find matches that are more varied. Were I to implement this in an EntityRuler pipe, I would need to have many patterns sit in its knowledge-base.

But finding one of these patterns isn’t enough. I want to ensure that the thing referenced is in fact a ship. Many of these terms could easily be toponyms, or entities that share the same spelling but mean different things in different contexts, e.g. the Ile de France, could easily be a GPE that refers to the area around Paris. General Hosey could easily be a PERSON. To ensure toponym disambiguation, I set up several contextual clues, e.g. the list of nautical terms. If any of these words appear in area around the hit, then the heuristics assign that token the label of SHIP. If not, it ignores it and allows later pipes or the machine learning model to annotate it.

ships_pattern = r"((S.S. |SS |The )*(Lieutenant Colonel James Barker|General Hosey|Pan Crescent|Marilyn Marlene|Winnipeg|Ile de France|Scythia|Aquitania|Empress of Britain|General A. W. Greely|General J. H. McRae|Empress of Scotland|General T. H. Bliss|New Amsterdam|Niagara|Henry Gibbs|Serpa Pinto|Mauretania|Cabo de Hornos|Julius Caesar|Ben Hecht|Sțrumah|Strumah|General Harry Taylor|General W.P. Richardson|Marine Jumper|Simon Bolivar|Pan York|Mauretania|Orduña|Wilhelm Gustloff|Orduna|General W.H. Gordon|Rakuyō Maru|Rakuyo Maru|Mouzinho|Saturnia|St. Louis|Saint Louis|Nyassa|Simon Bolivar|Queen Elizabeth|Exodus 1947|Dunera|Cap Arcona|Ernie Pyle|Hayim Arlozorov|Patria))"
@Language.component("find_ships")
def find_ships(doc):
    text = doc.text
    new_ents = []
    original_ents = list(doc.ents)
    nautical = ["ship", "boat", "sail", "captain", "sea", "harbor", "aboard", "admiral", "liner"]
    for match in re.finditer(ships_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        context = text[start-100:end+100]
        if any(term in context.lower() for term in nautical):
            if span is not None:
                new_ents.append((span.start, span.end, span.text))
            else:
                span = doc.char_span(start, end-1)
                if span is not None:
                    new_ents.append((span.start, span.end-1, span.text))
    for ent in new_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="SHIP")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_ships")
<function __main__.find_ships(doc)>

Let’s take a look and load in some data to see how this pipe functions. Remember, our goal is not to capture all ships, just ensure the ones we captured are true positives.

import glob
hits = []
files = glob.glob("data/new_ocr/*trs_en.txt")
all_data = {}
for file in files[:30]:
    all_hits = []
    with open (file, "r", encoding="utf-8") as f:
        print (file)
        text = f.read()
        doc = nlp(text)
        for ent in doc.ents:
            if ent.label_ == "SHIP":
                print (ent.text, ent.label_)
                print (text[ent.start_char-100:ent.end_char+100])
                print ()
data/new_ocr\RG-50.030.0001_trs_en.txt
data/new_ocr\RG-50.030.0002_trs_en.txt
data/new_ocr\RG-50.030.0003_trs_en.txt
data/new_ocr\RG-50.030.0004_trs_en.txt
data/new_ocr\RG-50.030.0006_trs_en.txt
Cap Arcona SHIP
to the middle of the Lübeck, Lübeck Bay and there was this gigantic ship which turned out to be the Cap Arcona. It was the Hamburg South American liner, 37,000 t ons luxury ship, luxury ship. And they put us up

Cap Arcona SHIP
 food Πanyway, somebody got this little ship and started to get it away from the, from the burning Cap Arcona. And I was one of the lucky ones who got on this little ship. And there were hundreds of people in 

Cap Arcona SHIP
May third by the British by mistake. So that's, that's the story of Πthat's the short story of the Cap Arcona. That was a, a fantastic luxury ship as, as we found out afterwards. It used to be. Now the ship go

Cap Arcona SHIP
hat nothing was known about it in, in other than the close-by German area. I think this book Œ that Cap Arcona book was, was researched and printed in, in Hamburg because all this Lübeck, Neustadt, Hamburg, thi

data/new_ocr\RG-50.030.0007_trs_en.txt
St. Louis SHIP
live, the answer came back...it is an internal matter. When the ship with eight hundred people, the St. Louis I believe it was, when that uh got in into the United States to Cuba, from the United States, they 

data/new_ocr\RG-50.030.0008_trs_en.txt
data/new_ocr\RG-50.030.0011_trs_en.txt
data/new_ocr\RG-50.030.0012_trs_en.txt
data/new_ocr\RG-50.030.0013_trs_en.txt
data/new_ocr\RG-50.030.0014_trs_en.txt
data/new_ocr\RG-50.030.0015_trs_en.txt
data/new_ocr\RG-50.030.0016_trs_en.txt
data/new_ocr\RG-50.030.0017_trs_en.txt
data/new_ocr\RG-50.030.0018_trs_en.txt
data/new_ocr\RG-50.030.0020_trs_en.txt
data/new_ocr\RG-50.030.0021_trs_en.txt
data/new_ocr\RG-50.030.0022_trs_en.txt
data/new_ocr\RG-50.030.0023_trs_en.txt
data/new_ocr\RG-50.030.0025_trs_en.txt
data/new_ocr\RG-50.030.0026_trs_en.txt
data/new_ocr\RG-50.030.0027_trs_en.txt
data/new_ocr\RG-50.030.0028_trs_en.txt
data/new_ocr\RG-50.030.0029_trs_en.txt
data/new_ocr\RG-50.030.0030_trs_en.txt
data/new_ocr\RG-50.030.0033_trs_en.txt
data/new_ocr\RG-50.030.0034_trs_en.txt
data/new_ocr\RG-50.030.0035_trs_en.txt
data/new_ocr\RG-50.030.0036_trs_en.txt
data/new_ocr\RG-50.030.0038_trs_en.txt

Success! File, data/new_ocr\RG-50.030.0006_trs_en.txt, referenced two ships, the Cap Arcona and the St. Louis. Now that we know we can capture ships in this manner, let’s try out the same principle on ghettos.

16.5. Creating a Pipe for Finding Ghettos

As of the time of pushing this notebook, August 13, 2021, I have yet to receive a comprehensive dataset of ghettos. In the near future, this pipe will be vastly improved, similar to the concentration camp pipe below. For now, this pipe functions precisely the same way our earlier ship pipe function. Here, we’re looking for the use of the word ghetto around one of these known cities that had ghettos. This is absolutely necessary because any of these cities could be a GPE in general. The use of the word ghetto within a small contextual window is a good heuristic for assigning this city the label of GHETTO over GPE.

ghetto_pattern = r"(Anykščiai|Anyksciai|Arad|Ashmiany|Babruĭsk|Babruisk|Balassagyarmat|Baranavichy|Barysaŭ|Barysau|Będzin|Bedzin|Bełżyce|Belzyce|Berdychiv|Berehove|Berestechko|Berezdiv|Berezhany|Berezne|Bershad'|Biała Podlaska|Biala Podlaska|Birkenau|Biała Rawska|Białystok|Bialystok|Biaroza|Bibrka|Bielsko-Biała|Biržai|Bitola|Blazhiv|Bobowa|Bochnia|Bolekhiv|Borshchuv|Boryslav|Boskovice|Brańsk|Bratslav|Brody|Brzesko|Buczacz|Budapest|Bus'k|Bychawa|Chashniki|Chrzanów|Chrzanow|Ciechanów|Ciechanow|Cieszanów|Cristuru Secuiesc|Czernowitz|Częstochowa|Czortków|Dąbrowa Górnicza|Dąbrowa Tarnowska|Damashėvichy|Daugavpils|Dokshytsy|Dombóvár|Dombrowa|Drohobycz|Drzewica|Dubrovytsia|Dzialoszyce|Dziarechyn|Dziatlava|Glebokie|Gol'shany|Góra Kalwaria|Gorodnaia|Gostynin|Gyöngyös|Hajdúszoboszló|Halushchyntsi|Halych|Hantsavichy|Haradnaia|Hatvan|Hlusk|Hlyniany|Homel'|Horodenka|Horokhiv|Hradzianka|Hrodna|Hvizdets'|Iaktoriv|Izbica Lubelska|Józefów|Kalisz|Kałuszyn|Kam'iane Pole|Kamin'-Kashyrs'kyĭ|Katowice|Kecskemét|Kelme|Kharkiv|Khmel'nyts'ka oblast'|Khmel'nyts'kyĭ|Khust|Kielce|Kisvárda|Kletsk|Kobryn|Kolbuszowa|Kolozsvár|Komarów-Osada|Kopychyntsi|Korets'|Košice|Kőszeg|Kovel'|Kozienice|Kraków|Kraśnik|Kretinga|Krośniewice|Krymne|Kryzhopil'|Kul'chyny|Kunhegyes|Kutno|Kysylyn|Ladyzhyn|Lakhva|Lask|Lęczyca|Lesko|Lida|Liepāja|Lipinki|Lithakia|Litin|Litzmannstadt|Liubavichi|Łomża|Lubaczów|Lubartów|Lublin|Łuck|Lwów|Lyubcha|Mahiliou|Maków Mazowiecki|Marcinkonys|Matejovce nad Hornádom|Mátészalka|Miechów|Międzyrzec Podlaski|Minsk|Mir|Miskolc|Modliborzyce|Mogilev|Monastyrok|Monor|Munkács|Nadvirna|Nagyvárad|Navahrudak|Novomyrhorod|Nowy Sącz|Nyíregyháza|Odessa|Oleyëvo-Korolëvka|Opatów|Opoczno|Opole|Opole Lubelskie|Orla|Orsha|Ostroh|Ostrowiec Świętokrzyski|Otwock|Ozarintsy|Ozorków|Pabianice|Papul|Parichi|Pechera|Pinsk|Piotrków Trybunalski|Płaszów|Płock|Plońsk|Praszka|Prienai|Prużana|Pruzhany|Przemyśl|Pułtusk|Radom|Radomyśl Wielki|Radun'|Rava-Rus'ka|Rawa Mazowiecka|Reghin|Ribnița|Riga|Rohatyn|Romanove Selo|Rozhyshche|Rudky|Rudnik nad Sanem|Rzeszów|Saharna|Šahy|Salgótarján|Sarny|Sátoraljaújhely|Schwientochlowitz|Senkevychivka|Sernyky|Sharhorod|Shchyrets'|Shepetivka|Shpola|Shumilino|Šiauliai|Siedlce|Siedliszcze|Sieradz|Sighetu Marmației|Skalat|Slobodka|Slonim|Slutsk|Smolensk|Sokołów Podlaski|Sokyrnytsia|Solotvyno|Soroca|Sosnowiec|Stalovichy|Stanislav|Stara Mohylʹnytsia|Starachowice|Starokostiantyniv|Stary Sącz|Stepan'|Stoczek Lukowski|Stolbëisy|Stolin|Sucha|Suchowola|Surazh|Švenčionys|Szarvas|Szczebrzeszyn|Szeged|Szolnok|Tarnogród|Tarnów|Telšiai|Terebovlia|Ternopol|Theresienstadt|Thessalonike|Timkovichi|Tlumach|Tolna|Tomaszów Mazowiecki|Torchyn|Trakai|Trebíč|Trnava|Tul'chyn|Tuliszków|Tyvriv|Uzda|Uzhhorod|Vác|Valozhyn|Velizh|Velykyĭ Bereznyĭ|Vilna|Vinnytsia|Vlonia|Volodymyr-Volyns'kyi|Vysokovskiy Rayon|Warka|Warsaw|Wisznice|Wrocław|Žagarė|Zamość|Zarichne|Zboriv|Zduńska Wola|Zhmerinka|Zhytomyr|Žiežmariai|Anyksciai|Arad|Ashmiany|Babruisk|Balassagyarmat|Baranavichy|Barysau|Bedzin|Bełzyce|Berdychiv|Berehove|Berestechko|Berezdiv|Berezhany|Berezne|Bershad'|Biała Podlaska|Biała Rawska|Białystok|Biaroza|Bibrka|Bielsko-Biała|Birzai|Bitola|Blazhiv|Bobowa|Bochnia|Bolekhiv|Borshchuv|Boryslav|Boskovice|Bransk|Bratslav|Brody|Brzesko|Buczacz|Budapest|Bus'k|Bychawa|Chashniki|Chrzanow|Ciechanow|Cieszanow|Cristuru Secuiesc|Czernowitz|Czestochowa|Czortkow|Dabrowa Gornicza|Dabrowa Tarnowska|Damashevichy|Daugavpils|Dokshytsy|Dombovar|Dombrowa|Drohobycz|Drzewica|Dubrovytsia|Dzialoszyce|Dziarechyn|Dziatlava|Glebokie|Gol'shany|Gora Kalwaria|Gorodnaia|Gostynin|Gyongyos|Hajduszoboszlo|Halushchyntsi|Halych|Hantsavichy|Haradnaia|Hatvan|Hlusk|Hlyniany|Homel'|Horodenka|Horokhiv|Hradzianka|Hrodna|Hvizdets'|Iaktoriv|Izbica Lubelska|Jozefow|Kalisz|Kałuszyn|Kam'iane Pole|Kamin'-Kashyrs'kyi|Katowice|Kecskemet|Kelme|Kharkiv|Khmel'nyts'ka oblast'|Khmel'nyts'kyi|Khust|Kielce|Kisvarda|Kletsk|Kobryn|Kolbuszowa|Kolozsvar|Komarow-Osada|Kopychyntsi|Korets'|Kosice|Koszeg|Kovel'|Kozienice|Krakow|Krasnik|Kretinga|Krosniewice|Krymne|Kryzhopil'|Kul'chyny|Kunhegyes|Kutno|Kysylyn|Ladyzhyn|Lakhva|Lask|Leczyca|Lesko|Lida|Liepaja|Lipinki|Lithakia|Litin|Litzmannstadt|Liubavichi|Łomza|Lubaczow|Lubartow|Lublin|Łuck|Lwow|Lyubcha|Mahiliou|Makow Mazowiecki|Marcinkonys|Matejovce nad Hornadom|Mateszalka|Miechow|Miedzyrzec Podlaski|Minsk|Mir|Miskolc|Modliborzyce|Mogilev|Monastyrok|Monor|Munkacs|Nadvirna|Nagyvarad|Navahrudak|Novomyrhorod|Nowy Sacz|Nyiregyhaza|Odessa|Oleyevo-Korolevka|Opatow|Opoczno|Opole|Opole Lubelskie|Orla|Orsha|Ostroh|Ostrowiec Swietokrzyski|Otwock|Ozarintsy|Ozorkow|Pabianice|Papul|Parichi|Pechera|Pinsk|Piotrkow Trybunalski|Płaszow|Płock|Plonsk|Praszka|Prienai|Pruzana|Pruzhany|Przemysl|Pułtusk|Radom|Radomysl Wielki|Radun'|Rava-Rus'ka|Rawa Mazowiecka|Reghin|Ribnita|Riga|Rohatyn|Romanove Selo|Rozhyshche|Rudky|Rudnik nad Sanem|Rzeszow|Saharna|Sahy|Salgotarjan|Sarny|Satoraljaujhely|Senkevychivka|Sernyky|Sharhorod|Shchyrets'|Shepetivka|Shpola|Shumilino|Siauliai|Siedlce|Siedliszcze|Sieradz|Sighetu Marmatiei|Skalat|Slobodka|Slonim|Slutsk|Smolensk|Sokołow Podlaski|Sokyrnytsia|Solotvyno|Soroca|Sosnowiec|Stalovichy|Stanislav|Stara Mohylʹnytsia|Starachowice|Starokostiantyniv|Stary Sacz|Stepan'|Stoczek Lukowski|Stolbeisy|Stolin|Sucha|Suchowola|Surazh|Svencionys|Szarvas|Szczebrzeszyn|Szeged|Szolnok|Tarnogrod|Tarnow|Telsiai|Terebovlia|Ternopol|Theresienstadt|Thessalonike|Timkovichi|Tlumach|Tolna|Tomaszow Mazowiecki|Torchyn|Trakai|Trebic|Trnava|Tul'chyn|Tuliszkow|Tyvriv|Uzda|Uzhhorod|Vac|Valozhyn|Velizh|Velykyi Bereznyi|Vilna|Vinnytsia|Vlonia|Volodymyr-Volyns'kyi|Vysokovskiy Rayon|Warka|Warsaw|Wisznice|Wrocław|Zagare|Zamosc|Zarichne|Zboriv|Zdunska Wola|Zhmerinka|Zhytomyr|Ziezmariai)"
@Language.component("find_ghettos")
def find_ghettos(doc):
    text = doc.text
    ghetto_ents = []
    gpe_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(ghetto_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        context = text[start-25:end+25]
        if "ghetto" in context.lower():
            if span is not None:
                ghetto_ents.append((span.start, span.end, span.text))
                
        else:
            if span is not None:
                gpe_ents.append((span.start, span.end, span.text))
    for ent in ghetto_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="GHETTO")
        original_ents.append(per_ent)
    for ent in gpe_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="GPE")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_ghettos")
<function __main__.find_ghettos(doc)>
import glob
hits = []
files = glob.glob("data/new_ocr/*trs_en.txt")
all_data = {}
for file in files[:5]:
    all_hits = []
    with open (file, "r", encoding="utf-8") as f:
        print (file)
        text = f.read()
        doc = nlp(text)
        for ent in doc.ents:
            if ent.label_ == "GHETTO":
                print (ent.text, ent.label_)
                print (text[ent.start_char-100:ent.end_char+100])
                print ()
data/new_ocr\RG-50.030.0001_trs_en.txt
Warsaw GHETTO
Q: I wanted to ask you if you saw the city, the Ghetto burning? A: The Jewish Ghetto, no, but I saw Warsaw. I was aware of the burning and I was aware that there was a Jewish uprising because it was full. E

Warsaw GHETTO
his whole experience. A: To record I could just tell you one thing. The worst for me was the Ghetto Warsaw. It was worse than hell. You couldn't imagine -- hell was worse. I can't imagine that the people co

data/new_ocr\RG-50.030.0002_trs_en.txt
Warsaw GHETTO
ve the strength to fight them. There was no way of fighting them. I mean sure there was uprising in Warsaw Ghetto, but there was no movement in Lodz Ghetto like this and _______________ saw to it. There was

data/new_ocr\RG-50.030.0003_trs_en.txt
data/new_ocr\RG-50.030.0004_trs_en.txt
data/new_ocr\RG-50.030.0006_trs_en.txt

As we can see, the pipe is working. We’ve grabbed three instances of Warsaw in data/new_ocr\RG-50.030.0001_trs_en.txt.

16.6. Create Pipe for Identifying a Person

For PERSON, we will be leveraging spaCy’s small model, but we can add some heuristics that will greatly improve the results. The heuristics here is any known salutation capitalized followed by a series of proper nouns. This RegEx “(?:[A-Z]\w+[ -]?)+)” allows us to grab all continuous capital words and then break when it encounters a non capital letter followed by a space. In English, these will always be PERSON entities.

people_pattern = r"((((Mr|Mrs|Miss|Dr|Col|Adm|Lt|Cap|Cpt|Fr|Cl|Cln|Sgt)\.)|(Frau|Herr|President|Rabbi|Queen|Prince|Princess|Pope|Father|Bishop|King|Cardinal|General|Liutenant|Colonel|Lieutenant Colonel|Private|Admiral|Captain|Sergeant|Sergeant First Class|Staff Sergeant|Sergeant Major|Corp Sergeant Major|Field Sergeant|Technical Sergeant|Corporal|Lance Corporal|Ensign|2nd Lieutenant|1st Lieutenant|Major|Hauptmann|Staff Captain|Oberst|Oberstlieutenant)) (?:[A-Z]\w+[ -]?)+)(the [A-Z]\w*|I\w*|X\w*|v\w*)*"
@Language.component("find_people")
def find_people(doc):
    text = doc.text
    match_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(people_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            match_ents.append((span.start, span.end, span.text))
                
        else:
            span = doc.char_span(start, end-1)
            if span is not None:
                match_ents.append((span.start, span.end, span.text))

    for ent in match_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="PERSON")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_people")
<function __main__.find_people(doc)>

16.7. Create Pipe for Identifying Spouses

Often times in historical documents the identity of people are referenced collectively. In some instances, such as those of spouses, this results in the name of the woman being attached to the name of her husband. The purpose of this SPOUSAL entity is to identify such constructs so that users can manipulate the output and reconstruct each individual singularly.

spousal_pattern = r"((Mr|Mrs|Miss|Dr)(\.)* and (Mr|Mrs|Miss|Dr)(\.)* (?:[A-Z]\w+[ -]?)+)"
@Language.component("find_spousal")
def find_spousal(doc):
    text = doc.text
    new_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(spousal_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            new_ents.append((span.start, span.end, span.text))
        else:
            span = doc.char_span(start, end-1)
            if span is not None:
                new_ents.append((span.start, span.end-1, span.text))
    for ent in new_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="SPOUSAL")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_spousal")
<function __main__.find_spousal(doc)>

16.8. Creating Concentration Camp Pipe

The correct identification of camp is one of the most important pipes in this pipeline. There are two concentration camp pipes. The first pipe looks at all known camps and subcamps and then looks for surrounding words to identify the context. The second pipe is less strict. It looks for all known main concentration camps without context. The reason for this is because sometimes the subcamps have the same names as frequently cited cities, e.g. Berlin or Neustadt. This is particularly true of the subcamps. The main camps, however, are frequently referenced to the camp itself. Both pipes are activated by default, but a user can deactivate one or other.

Throughout this pipe, you will see many functions that contain the name “getter”. These are custom functions that allow us to add special attributes to our entity spans. If you scroll down to the bottom of this section, you will see that we can use the HGC dataset for conentration camps to retrieve other salient information, such as the subcamp’s main camp, its longitude and latitude, opening date, closing date, etc.

def subcamp_getter(hit):
    hit = hit.text
    df = pd.read_csv("data/hgc_data.csv")
    subcamps = df.Main.tolist()
    camps = df.SubcampMattingly.tolist()
    i=0
    potential = []
    for c in camps:
        
        try:
            all_c = c.split("^")
            for c in all_c:
                c = c.replace("\(", "(").replace("\)", ")")
#                 if c == "Buna-Monowitz (Auschwitz III)":
#                     print (c)
                if hit.strip() == c.strip():
#                     print (hit, c)
                    if subcamps[i] not in potential:
                        potential.append(subcamps[i])
        except:
            AttributeError
        i=i+1
    return (potential)
def date_open_getter(hit):
    hit = hit.text
    df = pd.read_csv("data/hgc_data.csv")
    dates = df.Date_Open.tolist()
    camps = df.SubcampMattingly.tolist()
    i=0
    potential = []
    for c in camps:
        
        try:
            all_c = c.split("^")
            for c in all_c:
                if hit == c:
                    if dates[i] not in potential:
                        potential.append(dates[i])
        except:
            AttributeError
        i=i+1
    return (potential)
def date_closed_getter(hit):
    hit = hit.text
    df = pd.read_csv("data/hgc_data.csv")
    dates = df.Date_Close.tolist()
    camps = df.SubcampMattingly.tolist()
    i=0
    potential = []
    for c in camps:
        try:
            all_c = c.split("^")
            for c in all_c:
                if hit == c:
                    if dates[i] not in potential:
                        potential.append(dates[i])
        except:
            AttributeError
        i=i+1
    return (potential)
def latlong_getter(hit):
    hit = hit.text
    df = pd.read_csv("data/hgc_data.csv")
    lats = df.LAT.tolist()
    longs = df.LONG.tolist()
    camps = df.SubcampMattingly.tolist()
    i=0
    potential = []
    for c in camps:
        
        try:
            all_c = c.split("^")
            for c in all_c:
                if hit == c:
                    if lats[i] not in potential:
                        potential.append((lats[i], longs[i]))
        except:
            AttributeError
        i=i+1
    return (potential)
def hgc_id_getter(hit):
    hit = hit.text
    df = pd.read_csv("data/hgc_data.csv")
    ids = df.HGC_ID.tolist()
    camps = df.SubcampMattingly.tolist()
    i=0
    potential = []
    for c in camps:
        
        try:
            all_c = c.split("^")
            for c in all_c:
                if hit == c:
                    if ids[i] not in potential:
                        potential.append(ids[i])
        except:
            AttributeError
        i=i+1
    return (potential)
def camp_type_getter(hit):
    hit = hit.text
df = pd.read_csv("data/hgc_data.csv")
camps = df.SubcampMattingly.tolist()
subcamps = df.Main.tolist()
i=0
final_camps = []
for c in camps:
    if c != "nan" and c != "FALSE":
        if subcamps[i] != "nan" and subcamps[i] != "FALSE":
            try:
                if c.split()[0] != "":
                    c=c.replace("*", "")
                    for item in c.split("^"):
                        final_camps.append(item.replace("(", "\(").replace(")", "\)").strip())
            except:
                AttributeError
    i=i+1
    
final_camps.sort(key=len, reverse=True)
final_list = "|".join(final_camps)
strict_camps_pattern = r"("+final_list+")"
# print (strict_camps_pattern)
@Language.component("find_camps_strict")
def find_camps_strict(doc):
    text = doc.text
    camp_ents = []
    original_ents = list(doc.ents)
    context_terms = ["camp", "concentration", "labor", "forced", "gas", "chamber"]
    for match in re.finditer(strict_camps_pattern, doc.text):
#         print (match)
        start, end = match.span()
        span = doc.char_span(start, end)
        context = text[start-100:end+100]
        if any(term in context.lower() for term in context_terms):
            if span is not None:
#                 print (span)
                camp_ents.append((span.start, span.end, span.text))
    for ent in camp_ents:
#         print (ent)
        start, end, name = ent
        per_ent = Span(doc, start, end, label="CAMP")
        per_ent.set_extension("subcamp", getter=subcamp_getter, force=True)
        per_ent.set_extension("date_open", getter=date_open_getter, force=True)
        per_ent.set_extension("date_closed", getter=date_closed_getter, force=True)
        per_ent.set_extension("latlong", getter=latlong_getter, force=True)
        per_ent.set_extension("hgc_id", getter=hgc_id_getter, force=True)
        
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
camps_pattern = r"(Alderney|Amersfoort|Auschwitz|Banjica|Belzec|Bergen-Belsen|Bernburg|Bogdanovka|Bolzano|Bor|Breendonk|Breitenau|Buchenwald|Chelmno|Dachau|Drancy|Falstad|Flossenburg|Fort VII|Fossoli|Grini|Gross-Rosen|Herzogenbusch|Hinzert|Janowska|Jasenovac|Kaiserwald|Kaunas|Kemna|Klooga|Le Vernet|Majdanek|Malchow|Maly Trostenets|Mechelen|Mittelbau-Dora|Natzweiler-Struthof|Neuengamme|Niederhagen|Oberer Kuhberg|Oranienburg|Osthofen|Plaszow|Ravensbruck|Risiera di San Sabba|Sachsenhausen|Sajmište|Salaspils|Sobibor|Soldau|Stutthof|Theresienstadt|Trawniki|Treblinka|Vaivara)(-[A-Z]\S+)*"
@Language.component("find_camps")
def find_camps(doc):
    text = doc.text
    camp_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(camps_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            camp_ents.append((span.start, span.end, span.text))
    for ent in camp_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="CAMP")
        per_ent.set_extension("subcamp", getter=subcamp_getter, force=True)
        per_ent.set_extension("date_open", getter=date_open_getter, force=True)
        per_ent.set_extension("date_closed", getter=date_closed_getter, force=True)
        per_ent.set_extension("latlong", getter=latlong_getter, force=True)
        per_ent.set_extension("hgc_id", getter=hgc_id_getter, force=True)
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_camps_strict")
nlp.add_pipe("find_camps")
<function __main__.find_camps(doc)>
import glob
hits = []
files = glob.glob("data/new_ocr/*trs_en.txt")
all_data = {}
for file in files[1:2]:
    all_hits = []
    with open (file, "r", encoding="utf-8") as f:
        print (file)
        text = f.read()
        doc = nlp(text)
        for ent in doc.ents:
            if ent.label_ == "CAMP":
                print ((ent.text, ent.label_, ent._.subcamp, ent._.date_open, ent._.date_closed, ent._.latlong, ent._.hgc_id))
                print (text[ent.start_char-100:ent.end_char+100])
                print ()
data/new_ocr\RG-50.030.0002_trs_en.txt
('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
ed the death, so we had this extra little food and an extra loaf of bread, which later I left if in Auschwitz because we're still hoarding and we're still afraid to let go and eat everything a once. But the ex

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
e expected that they were going to take him on an inspection of camp and he's going to help them in Auschwitz. And there are three theories what happened to him. One was that some people from Lodz Ghetto kille

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
 were all kinds of rumors but no one knew. Q: Had you heard about this before? A: We didn't hear of Auschwitz. We heard about bad things happening, but the only time that I knew something was happening when th

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
 mention it in my book. So, when we got -- what were we talking about? Q: The cattle cars, going to Auschwitz. A: Yes, the cattle cars, going to Auschwitz. We arrived at Auschwitz, and it was dark in the cattl

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
at were we talking about? Q: The cattle cars, going to Auschwitz. A: Yes, the cattle cars, going to Auschwitz. We arrived at Auschwitz, and it was dark in the cattle cars. All of the sudden the light was kind 

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
 Q: The cattle cars, going to Auschwitz. A: Yes, the cattle cars, going to Auschwitz. We arrived at Auschwitz, and it was dark in the cattle cars. All of the sudden the light was kind of blinding. They lined u

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
o of us. Q: Was there a general hysteria? A: No. Q: People just weren't --? A: There's a by word of Auschwitz "shnell (ph)" Everything they did you had to do fast, 22 fast, fast. You had no time to regroup, yo

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
have a barrack for us. They'll send us to the latrine. I don't know if you know what the latrine in Auschwitz looked. They were concrete benches with holes that could accommodate 500 people at once. There were

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
ing as toilets. Well, that's where we spent the night, in the toilets. Well, I don't have to repeat Auschwitz stories. We stood up from 5:00 in the morning or 6:00 in the morning and then they put us in the ba

('Bergen-Belsen', 'CAMP', ['Bergen-Belsen'], ['03/21/1943'], ['04/15/1945'], [(52.758899, 9.907099)], ['02-0065-0'])
e I wasn't moving fast enough, there was this famous ___________ Erma __________, she later came to Bergen-Belsen, blonde, gorgeous cool and she had this big whip and she smacked me with a whip. Because as I said,

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
 to BergenBelsen. Q: Why don't you tell me how that came about? A: About two weeks after we were in Auschwitz, they lined up, had another selection. They took us to the showers, gave us some clothes, and they 

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
ection. They gave us a bath, and they put us on the train. This was the time I really don't know if Auschwitz had two stations or not because this is the only time and I went back to Auschwitz, I remember, we 

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
ly don't know if Auschwitz had two stations or not because this is the only time and I went back to Auschwitz, I remember, we went through the main camp of Auschwitz, because we were in ___________ and this wa

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
se this is the only time and I went back to Auschwitz, I remember, we went through the main camp of Auschwitz, because we were in ___________ and this was, I saw the red buildings, the permanent buildings. I s

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
the permanent buildings. I saw the band playing. So, I had a feeling they had some other station in Auschwitz proper which they loaded us. They unloaded the cars somewhere else, and they put us there. They put

('Bergen-Belsen', 'CAMP', ['Bergen-Belsen'], ['03/21/1943'], ['04/15/1945'], [(52.758899, 9.907099)], ['02-0065-0'])
 I was in Europe and we traveled later, I was very reluctant to take a train. We finally arrived at Bergen-Belsen about maybe the third day. All the passing and stopping we passed, I think, Dresden. We saw someone

('Bergen-Belsen', 'CAMP', ['Bergen-Belsen'], ['03/21/1943'], ['04/15/1945'], [(52.758899, 9.907099)], ['02-0065-0'])
we were going, what we're doing. Anyway, they took us, we came to BergenBelsen, and until that time Bergen-Belsen was not a concentration camp. They had various camps of various entities, prisoner of war camp, but

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
 is shaved?" They couldn't understand that. There were SS but they had no idea about the horrors of Auschwitz or the other camps. They were just half way decent people in the beginning. The food we got was par

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
camps. They were just half way decent people in the beginning. The food we got was paradise next to Auschwitz. Fairly decent soup, we got some kind of coffee in the morning and a nice piece of bread and at nig

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
t time there was 3,000 people in that compound, because there were other groups. Q: All coming from Auschwitz? A: I think so. I think they were coming from Auschwitz because the other ones were Hungarians. Can

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
there were other groups. Q: All coming from Auschwitz? A: I think so. I think they were coming from Auschwitz because the other ones were Hungarians. Can I tell you something interesting about -- by that time,

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
something, hope. To do something like this in such circumstances was unbelievable, especially after Auschwitz where we were so controlled and here we could attend this service. Q: About the camp, were you isol

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
ngs were getting bad, more transports. I believe this was the time they brought the commandant from Auschwitz and he was cruel. Q: What was his name? A: I think Horss. And ____________ that she's the head and 

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
nd ____________ that she's the head and she was blonde and some others. I mean this nasty crew from Auschwitz came. 36 Q: Do you know about what time, what year that was? A: This was in '44, had to be around w

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
s? A: This was in '44, had to be around winter of '44, because it was getting cold when we lived in Auschwitz, when we had those surgeries. Also, when they were bringing the soups to the hospital, there were a

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
nk Poland by that time was clean. They finished with those. Q: They weren't necessarily coming from Auschwitz? A: No, no those people were coming with their own clothes and whatever the camp wore. They were no

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
, no those people were coming with their own clothes and whatever the camp wore. They were not from Auschwitz. They were not from camps. They were wherever they lived in. Anyway, then it was getting almost lig

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
d to him. He disappeared. Q: You mentioned a name "Kramer." Isn't Kramer someone who also came from Auschwitz and replaced Horss? A: I don't know whether he replaced Horsss. I don't know. I think Horss was cau

('Bergen-Belsen', 'CAMP', ['Bergen-Belsen'], ['03/21/1943'], ['04/15/1945'], [(52.758899, 9.907099)], ['02-0065-0'])
placed Horss? A: I don't know whether he replaced Horsss. I don't know. I think Horss was caught in Bergen-Belsen. I really don't know. Q: This Kramer then was head of the camp? A: I know there was someone by the 

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
culated and toward the end there was a rumor they were going to evacuate the camp, like they did in Auschwitz. There was a march. There was a group that went on trains, and I think the trains were bombed, and 

('Bergen-Belsen', 'CAMP', ['Bergen-Belsen'], ['03/21/1943'], ['04/15/1945'], [(52.758899, 9.907099)], ['02-0065-0'])
, and they were just very good, trying to help the people and take care of them. The British burned Bergen-Belsen as they emptied it. They left one building standing and I don't know when there was a ceremony and 

('Bergen-Belsen', 'CAMP', ['Bergen-Belsen'], ['03/21/1943'], ['04/15/1945'], [(52.758899, 9.907099)], ['02-0065-0'])
y raised the British flag over it because they never raise a British flag over a place as horrid as Bergen-Belsen. Q: Did they burn it just for the horror of it, or did because they wanted to burn off the disease?

('Auschwitz', 'CAMP', ['Auschwitz'], ['06/14/1940'], ['01/27/1945'], [(50.026199, 19.204099)], ['01-0032-3'])
hey did it deliberately and gradually. One step at a time, one humiliation. The fact when we got to Auschwitz and they undressed us and we had to march in front of men. And the women, there were religious wome

16.9. Creating Revolutionary Groups Pipe

The purpose of this pipe is to find known Revolutionary Groups. Again, this pipe is not an EntityRuler because I intend to do a few extra things with it in the future beyond the limitations of the EntityRuler.

groups_pattern = r"(Ethnikon Apeleutherotikon Metopon|Weisse Rose|Rote Kapelle|Affiche rouge|Edelweisspiraten|White Rose|Bielski|Nekamah|Voroshilov|OEuvre de secours aux enfants|Union des juifs pour la résistance et l'entraide|Zorin Unit|Komsomolski|Fareynikte|Korzh|Zhukov|Budenny|Parkhomenko|Sixième)((-)*[A-Z]\S+)*( (Brigade|brothers|group))*"
@Language.component("find_groups")
def find_groups(doc):
    text = doc.text
    camp_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(groups_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            camp_ents.append((span.start, span.end, span.text))
    for ent in camp_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="GROUP")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_groups")
<function __main__.find_groups(doc)>

16.10. Creating a Places Pipe

The ML model will capture place fairly well. Nevertheless, if you can write a simple rule, write a simple rule.

city_pattern = r"(?:[A-Z]\w+[ -]?)+, (Germany|Poland|England|Russia|Italy|USA|U.S.A.|United States|United States of America|America|United Kingdom|France|Spain|Ukraine|Romania|Netherlands|Belgium|Greece|Portugal|Sweden|Hungary|Austria|Belarus|Serbia|Switzerland|Bulgaria|Denmark|Finland|Slovakia|Norway|Ireland|Croatia|Moldova|Bosnia|Albania|Estonia|Malta|Iceland|Andorra|Luxembourg|Montenegro|Macedonia|San Marino|Lichtenstein|Monaco)"
country_pattern = r"(Germany|Poland|England|Russia|Italy|USA|U.S.A.|United States|United States of America|America|United Kingdom|France|Spain|Ukraine|Romania|Netherlands|Belgium|Greece|Portugal|Sweden|Hungary|Austria|Belarus|Serbia|Switzerland|Bulgaria|Denmark|Finland|Slovakia|Norway|Ireland|Croatia|Moldova|Bosnia|Albania|Estonia|Malta|Iceland|Andorra|Luxembourg|Montenegro|Macedonia|San Marino|Lichtenstein|Monaco)"
@Language.component("find_places")
def find_places(doc):
    text = doc.text
    new_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(city_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            new_ents.append((span.start, span.end, span.text))
        else:
            span = doc.char_span(start, end-1)
            if span is not None:
                new_ents.append((span.start, span.end-1, span.text))
    for ent in new_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="GPE")
        if per_ent.text.split(",")[0] not in city_pattern:
            original_ents.append(per_ent)
            
    for match in re.finditer(country_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            new_ents.append((span.start, span.end, span.text))
        else:
            span = doc.char_span(start, end-1)
            if span is not None:
                new_ents.append((span.start, span.end-1, span.text))
    for ent in new_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="GPE")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_places")
<function __main__.find_places(doc)>

16.11. Creating a Geography Pipe

general_pattern = r"([A-Z]\w+) (River|Mountain|Mountains|Forest|Forests|Sea|Ocean)*"
river_pattern = "(the|The) (Rhone|Volga|Danube|Ural|Dnieper|Don|Pechora|Kama|Oka|Belaya|Dniester|Rhine|Desna|Elbe|Donets|Vistula|Tagus|Daugava|Loire|Tisza|Ebro|Prut|Neman|Sava|Meuse|Kuban River|Douro|Mezen|Oder|Guadiana|Rhône|Kuma|Warta|Seine|Mureș|Northern Dvina|Vychegda|Drava|Po|Guadalquivir|Bolshoy Uzen|Siret|Maly Uzen|Terek|Olt|Vashka|Glomma|Garonne|Usa|Kemijoki|Great Morava|Moselle|Main 525|Torne|Dalälven|Inn|Maritsa|Marne|Neris|Júcar|Dordogne|Saône|Ume|Mur|Ångerman|Klarälven|Lule|Gauja|Weser|Kalix|Vindel River|Ljusnan|Indalsälven|Vltava|Ponoy|Ialomița|Onega|Somes|Struma|Adige|Skellefte|Tiber|Vah|Pite|Faxälven|Vardar|Shannon|Charente|Iskar|Tundzha|Ems|Tana|Scheldt|Timiș|Genil|Severn|Morava|Luga|Argeș|Ljungan|Minho|Venta|Thames|Drina|Jiu|Drin|Segura|Torne|Osam|Arda|Yantra|Kamchiya|Mesta)"
@Language.component("find_geography")
def find_geography(doc):
    text = doc.text
    river_ents = []
    general_ents = []
    original_ents = list(doc.ents)
    for match in re.finditer(river_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            river_ents.append((span.start, span.end, span.text))
    for match in re.finditer(general_pattern, doc.text):
        start, end = match.span()
        span = doc.char_span(start, end)
        if span is not None:
            general_ents.append((span.start, span.end, span.text))       
            
#     all_ents = river_ents+general_ents       
    for ent in river_ents:
        start, end, name = ent
        per_ent = Span(doc, start, end, label="RIVER")
        original_ents.append(per_ent)
        
    for ent in general_ents:
        start, end, name = ent
        if "River" in name:
            per_ent = Span(doc, start, end, label="RIVER")
        elif "Mountain" in name:
            per_ent = Span(doc, start, end, label="MOUNTAIN")
        elif "Sea" in name:
            per_ent = Span(doc, start, end, label="SEA-OCEAN")
        elif "Forest" in name:
            per_ent = Span(doc, start, end, label="FOREST")
        original_ents.append(per_ent)
    filtered = filter_spans(original_ents)
    doc.ents = filtered
    return (doc)
nlp.add_pipe("find_geography")
<function __main__.find_geography(doc)>

16.12. Seeing the Pipes at Work

Now that we have assembled all these pipes into a pipeline, let’s see how they perform on a real testimony.

import glob
from spacy import displacy
hits = []
files = glob.glob("data/new_ocr/*trs_en.txt")
all_data = {}
for file in files[5:6]:
    all_hits = []
    with open (file, "r", encoding="utf-8") as f:
        print (file)
        text = f.read().replace("(ph)", "")
        while "  " in text:
            text =  text.replace("  ", " ")
        doc = nlp(text)
        colors = {"CAMP": "#FF5733", "GHETTO": "#1F9D12", "SHIP": "#557DB4", "SPOUSAL": "#55B489", "GPE":"#17B4C2", "RIVER": "#9017C2", "MOUNTAIN": "#878787", "SEA-OCEAN": "#0A6DF5", "FOREST": "#1F541D"}
        options = {"ents": ["CAMP", "GHETTO", "SHIP", "SPOUSAL", "GPE", "RIVER", "MOUNTAIN", "SEA-OCEAN", "FOREST"], "colors":colors}
        displacy.render(doc, style="ent", jupyter=True, options=options)
data/new_ocr\RG-50.030.0007_trs_en.txt
HARRY ALEXANDER April 25, 1991 Q: Please tell us your full name? A: My name is Harry Alexander. Alex for short. Q: And where and when were you born? A: I was born July 12, 1921 in Leipzig, Germany GPE . Q: Could you tell us a little bit about your family life? A: Yes. We were seven children, four boys, three girls, my mother, my father. We, I would say, upper middle class Jewish family. We had a good life. We had want for nothing. And uh we was just happy all around, just a happy family life. We had a good family life. Q: What kind of work did your father do? A: My father was a chemical engineer. My mother was a housewife. Uh we all went to school and uh we had a very comfortable existence. It was good living in those days in in Germany GPE . It was really was a good life, a good wholesome life. And uh we had a good social life and uh my brothers were...I'm the youngest of the boys and had three little sisters, and uh we all loved one another. Everything was fine, until...until uh 1938 I really did not believe...comprehend what was going on. I remember my mother keep saying, don't be outside too long. Come on home and and uh when you go to the schul , the temple, come right back home and we kept the holidays and we had a all around good life and uh people were talking and uh...about the Nazis. We saw the brown shirts and the SS and and and uh sure we weren't allowed to go to parks, but we...I all...failed to understand the significance of the whole thing. Uh we weren't allowed to uh go to movies. We weren't' allowed to go to the zoo...Jews uh not allowed. We weren't allowed to uh go shopping in stores. We used to go to a store to buy something. We had to wait till everybody was served. We were always the last one to get what we want, if we were lucky enough to get it. And by by the time I was eleven, twelve years old, I was thrown out of school. Well, there there was a Jewish school in my uh hometown, in Leipzig , but there were so many Jewish kids that they didn't have enough room for all the kids. So we only could go to school maybe two hours a week or three hours a week in shifts. And we had all this...I took stride. I just didn't...I really didn't understand this whole thing, and nobody else really explained anything to us. They would try to keep it away from us...until uh 1938. The week before Kristallnacht when they start to arrest all the Jews of Polish background origin, they deported them. Uh the children were in school and the fathers were at their place of work and the mothers were in in the houses cooking and things, and just ______ were arrested, one after another, put to...not not given time uh to pack anything. Took them to trains...the father in one train, the mother in the other train. 2 Nobody knew where anybody was and the children came home from school. The place was padlocked. They sat outside, didn't know what was happening to them. Then they grabbed the children, put them on another train until they they worked it out, what was happening and everybody started running for the em...consulate, the embassies, because the embassies...they couldn't get in. They were safe there. So they went to the French Embassy. They went to the American Embassy. They went to the Russian Embassy. They went to the Danish, Norwegian, Swiss...anything that was near. So our job was to bring food to these people and take messages, because if the wife was in the Danish Embassy and the husband was in the in the French Embassy, we just took messages back and forth trying to seek the families out to tell them that their loved ones were alright. That went on for a couple of days, and uh the night...the 8th of November...I believe it was the 9th or the 8th...8th or 9th of November I believe...I never forgot...the uh the uh Kristallnacht. That is when I really really was hit with the uh with the happenings there. They came to our house. They broke everything up and like that in every other house. They were raping the young girls in the streets and and dragging men by their beards through the streets and burning down the temples and and and and and threw...I remember they threw all our furniture and dishes and everything out the window and and and uh beat us up. They came in...the Hitler youth...with their daggers, blood and honor, and punks, bands of punks and they're beating everybody up, taking people to concentration camps, arresting people and it was just a terrible, terrible night. It started about three o'clock in the morning. That night they arrested my brothe r Paul. My brother uh Bernie , he had left. Of course, there was still home...I was still home, my three sisters. We were hiding in one of the beds but they dragged us out, beat up my mother and the next day it was quiet. The smoke was still coming out of the buildings of the temples and the schules and houses and there was still blood in the street. Now I knew...now I knew...something terrible is happening. But I didn't know what to do about it. I had my mama. I had my mother. She was always taking care of us. (Sigh) Well, my brother Paul to Buchenwald CAMP . My other brother went to Sachsenhausen CAMP . They were taken, and a few days later they came to our house from a Jewish organization and they said we can save one child. Which one is going to live and which ones are going to die? (Sigh) My mother had to make a choice and by the time the choice was made and everything, the other one was arrested. Was just left me and my three little sisters. That was it. A couple of months passed and my mother sent me out for a loaf of bread and I never came back. A car came screeching to a halt...are you a Jew? I was sixteen years old...fifteen, sixteen years old. I said yes. I didn't know what to say. These were mean looking men. They put me into the car and I found myself in prison. No explanation was given. None was asked. I remember the cell, cell number ten. My mother wondered what happened to me...send me out for a loaf of bread. I was in jail about...I would say about two months, three months...till May, approximately May. Suddenly they let me go free. I had to sign a paper that I was going to leave Germany GPE within forty-eight hours...they will let me go, and that I was treated well. Again I didn't know what I was signing. They told me to sign. I signed. And I went home. I remember to this day...my mother stay...stood at the door of our apart...our house. She had ten marks...that was all we were allowed to take. Ten marks. She must have had this in cash laying around because all our money had been confiscated by the 3 state, and we had nothing left and she said to me, you have to leave home. Don't even go into the house. You have to leave home right now. Here is ten marks. She says go to Italy GPE . Italy GPE was the only country that had open borders because they are friends with ______. Hitler was the leader. They were friends. They had no closed border. And that was the only country that had an open border for Germans. I had no passport. I had no papers. I had nothing. Here...take the train, she says. Here is a card, a railroad ticket I got for you. Take the train and go into Italy GPE and just one by one...don't ever look back. So I said to her where are my other two sisters. I had three sisters. She said they're on their way to Poland GPE . I sent them with some people, with some friends to Poland GPE . Only Toni , the little one, she had in her arm. Must have been about...I don't know...four, three, four years old. So I said goodbye...I was a good boy so I said goodbye. Mamma said go...I go. And uh that is when my life started. I went to the railroad station. I took a train into Italy GPE . Q: Where was your father? A: Gone. Q: When did...where did he go? A: He went into a camp. Q: When he was taken there...Kristallnacht? A: Yeah. Yeah. Now my mother was alone with my little sister. How my mother ever got me out of that jail I will never know to this day. I am sure she must have bribed a lot of people to get me out. But she saved my life. So where does one go? One goes to Milano , Italy GPE . In Milano, arriving in Milan...we had no place to go. I wasn't the only one. There were hundreds...men, women, children...anyone who could themselves, they went down there. (Sigh) We had no food to eat. We had no place to sleep. We slept on park benches, in doorways, in parks, and we _____ like a hole, like rats. So the Jews of Milan, they got together and they opened up soup kitchens and they got organized and so at least we could have one meal a day, somewhat of a meal a day. That was done for a while, a week, two weeks. Then the Germans found out that the Jews are treated to some...to a meal a day, and they they made arrangements with the Italian authorities because we told the Italians that we were tourists, otherwise you couldn't get in. We were tourists. So they let us in. And now they said if you're tourists, then you've got to have money and if you have money, you don't need any soup kitchens, and if you're not tourists, then we have to send you back where you came from. Made a lot of sense to them...didn't make any sense to us though. They closed up the soup kitchens. That was the end of food. _____ more than one meal a day. So another few days and uh like I always was, I used to climb into the tree, some backyards, and pluck some fresh figs. (Laughter) I...I stole some olives from trees, but there was _____ to steal. It was edible and to stay alive. Then the word got around that France GPE will let anybody in. The men have to go to the French Army, 4 because war was eminent, and the women had to volunteer for the French Red Cross. We said wonderful. We all went to the French Embassy, and we all got that...the one day flag they call it. We got a visa to enter France GPE that was good for 24 hours only. Well, there was only train going to the French border because the war broke out quite a few weeks after that. We had to get that train and they gave us a train ticket so we went for that train. Now what that train looked like you will never know. They were hanging out of the windows. They were laying on the roofs...inside...outside. It was unbelievable. They...they...they held children out out the windows. There was room for them. They were suffocating inside. There must have been a thousand people or maybe fifteen hundred people on a on a train that is good for three hundred people. The train took off. It couldn't go very fast because people were on the roofs. They were hanging out windows. Outside...inside and they were hanging on their fingernails. People fell off, actually fell off the train while the train was moving so the train conductor didn't go to fast because he saw what was going on. By the time we got to the Italian/French border, it was too late. The French had closed the border and there's a little piece of land called no man's land and the Italians wouldn't let us back. And that was the end of that. So we laid on that little piece land...there was a fishing village. ______. St. Reno ____...it's right on the border. ______. It's right on the border, and these old fishermen with these old boats...they saw a way to make money. Now these were impoverished people that never had anything except to catch a few fish now and then. So they said to us, we'll take you around the bent...it's a big rock...and you had to go around the bent onto the beaches of France GPE if you give us everything you have and uh pay us whatever you can pay, and we'll load up the boats. We'll take you. But what they didn't tell people, that the boats were old and that the breakers in the ocean were very strong onto the onto the uh ____, unto the stone, unto the uh onto the mountain. The mountain came back out of the ocean. The waves were high and very strong and very uh uh dangerous. So as greedy as they were, they overloaded the boats. Some people had a wedding band, an engagement ring, maybe a few francs, maybe a few lire, maybe a little piece of jewelry that they saved, that they smuggled out. They overloaded. They gave them everything they owned. They overloaded the boats. They went out a little bit, and the boats broke in two. They capsized. They broke in two. The fishermen swam back. He has been fishing...he has been swimming since he was two years old, so there were people that couldn't swim. Now you tell me...who do you save when you're a father and and your little girl says Daddy, save me, and your little boy says Papa, save me. And your wife is drowning. Who do you save...and you can't swim, and the waves are high. The next day the the bodies came washing ashore. And there were two boats out there, two uh gun boats. A French one and an Italian gunboat. And the the personnel on these boats, they were looking out and they saw the whole thing. Nobody lifted their finger to help. The screams of help, please help, help my children, help my wife, uh save us...still ring into my ears to this day. Nobody cared. I met somebody there that knew my family and they said to me, you know about your brother Paul. I says what. He got out. He went to Belgium GPE . The word got out. He was married. He had a twenty year old wife and he had a one year old baby. I said you know what happened to Judy . Have you heard and had any word. He says yes. He tried to smuggle them out into Belgium GPE . He paid smugglers to bring them out. 5 The smugglers took all their...what they owned in the middle of the woods and abandoned them. The baby started to cry. The Germans got them and hang them. They hang ___ from the ___ wife and baby next to her. They were found like that. I said anything you know about my brother Paul. He says yes. He went half crazy. So the word got out and he went into France GPE . That's the last we heard of him. Because here and there, here and there, you found people that knew something about somebody. (Sigh) Now we're stuck there on this piece of land there, and we had to...well not much of a chance to run to...to one side you had the French with their guns looking at us, and on the other side you had the Italians with guns looking at us, and to the front you had the ocean. There was nothing you could do with the ocean and in the back of you, you had a mountain. The highest mountain we ever saw was the garbage dump in our city. I never saw a mountain in my life before. How do you get out? A little boy hung on to me. He was twelve years old. His name was Heinz . He got lost, someplace along the the running, he got lost from his family. He says he wants to go with me. I said hang on, and then a few days passed by and the only help you got was from a few monks that used to come and used to give us a little milk and a little piece of bread, a few figs, a few uh uh olives and things. It was a pitiful sight. And yet I wasn't really worried. I just absorbed all this like matter of fact, and then said well, I'm going to do something about this. And I looked around, tried to make up my mind which way to go, and uh I said wouldn't the best approach be just go right up to the French border, crawl through the barbed wire there, and get inside. Now wouldn't that be just just a crazy thing to do. Said to Heinz, look...we can't take the ocean. We can't go back to Italy GPE . We can't officially go to France GPE . We can't go the mountain. We'll never...never make it. I have an idea. Let's do the crazy thing that nobody expects us to do. He says, what? I said let's go right through the barbed wire into...let's go right into France GPE . He said, you're crazy. I said yes, that's why...that's what everybody thinks...I'm crazy. Let's do the crazy thing. That's what we did. I said we're going to die. We're going to starve to death here, and there must have been about a thousand, maybe fourteen hundred people lying on this ground. I don't know ever what happened to them. I don't...never found out what happened to them. We went through the wire, and I said you just follow me. What happen will happen. What have we got to loose? And I wasn't worried really, not really. I was maybe too young to worry about things and too ignorant. (Laughter) We did that. Coming out on the other side I was looking up and I looked into the barrel of a gun. There was a French guard there and border guards there. He looked down at me. I looked up at him, and Heinz stopped crawling and I stopped crawling, and he had the gun aimed at us, so I said to him in German...shoot me but let him live. He's only twelve years old. So it was that somebody something in French. I could not speak it. He...he called something back in French, and he says...he spoke broken German...he said to me, see that train over there. I said yes. Get in it. It's going to go to Nice, into France GPE . ____________ uncivilized person. We went in. We were very tired and we fell asleep, two of us. Dirty, bloody, hungry, thirsty, cold...went to sleep. We woke up in Nice. We got out in Nice and we dragged ourselves through the streets. I spoke no French. We had...we'd never been in a French country. We didn't know what France GPE was all about, and we went to a doorway and went asleep, and they arrested us, the gendarmes. That is when I first met the gendarmes, the guard du 6 mobile , and let me assure you the gendarmes and the guard du mobile, they are not the gentlest of people. They really are nasty dogs. They grabbed us. They called us spies, ______ spies there. I was a spy and they threw us in jail. Now in France GPE if you're caught about illegally, they...that's the law. You'll get one month in jail, before they even talk to you. I learned to understand what jail is in France GPE . Jail is a dirty cell with roaches and rats and and you get a drink of water a day and a piece of bread full of ants. In order to eat the bread full of...and you knock out the ants as best you can...you break off a piece of bread and throw it in the corner, let the rats fight over it while they let you eat...quickly eat the other piece. That's is jail in France GPE . After one day, took me out and they put me into a open car, like they called that the ______ commando, like with eight people could sit there, like a long open van or something...I don't know what you call it. Eight guard du mobile, eight of them...they put me in chains. My...my legs and my arm in chains, and it took eight guards du mobile...guard du mobile is the French police and gendarmes...to guard me, after they chased me down, after I was already down to nothing from lack of food and lack of sleep and and and I had no more tears left to cry. I don't know what happened to Heinz. And they took me out. Nobody told me where we were going. Nobody told me anything. We went to Entibe , a place called Entibe. Entibe is outside Nice, a concentration camp. They call it internment camp, but don't you believe it. Don't you believe internment camps. Resettlement camps. Don't you believe it. These are concentration camps, in the sense of the word...concentration camps with all their pitfalls, all their filth and all their dirt and and and and and and all their...all their nastiness and brutality. There was no place for us to sleep. We slept on floors under the open sky. Was ____ maybe two thousand people in there. They pulled up in front of the gate, opened up the gate, took off the chains, grabbed me and threw me into the camp. And I fell down on my face into into the dirt and that was it. They did this. That's what they did. I was in that camp, oh must have been well maybe six weeks, maybe two months and they took us back on transports on trucks and they put us into another camp which was called ________, __________. That is __________ Marseilles . Q: In in the first camp, did you have to do any labor or any work? A: No no. Q: What did you do during the day? A: Just hang around, nothing to do. Tried to stay alive. Tried to fight off the lice and the and the rats and the and the and stay away from the guards who would brutalize you. Uh some of them would uh try to get young boys and uh you know...I don't have to tell you. There was homosexuality there almost rampant by our own people and the guards. And there were always people that would uh sort of protect us. We were so innocent. I never heard the word homosexuality. I never knew what it was. I'd never had anything to do with it, but uh I met a friend of mine. He came from my home town. He was ten years older, a _____. We got to talking. He says uh we come from the same town and uh they had a store across the street where we lived and we sort of got together. He sort of kept 7 me...you know, kept an eye on me. Chased people away. When the other camp ______, ______ was all brick factory. They had the catacombs and they would lock us into the catacombs like little tunnels where they used to dry the bricks, like ovens. There was this straw on the floor. They would chase us in there by the hundreds and lock the door, a big iron door, and that was it. And we were lucky enough...was total darkness in there...when they stayed they put a few little bulbs and they are flickering in there and that is what...there we were. That is what we did. Honest to God...that is how we laid on the straw and and and and just uh tried to exist. After a couple of months of that kind of life...well, they took us out once a day for exercise. They let us walk around the yard. Q: What about food? A: (Sigh) Well, food...uh I wasn't a big eater anyhow in those days. (Laughter) I never thought about food that much but the food was very bad there. They gave us a cup of coffee a day, black water, and gave us some...a a half a liter of soup at lunch time and a half a liter of soup in the evening and if you are...were lucky you found a rotten onion in it or a rotten potato or a couple roaches. (Laughter) There were...this was about uh...well, there was darkness in there so you didn't really see it, you know, so it didn't matter too much. We existed. They gave you a piece of bread, gave you seventy-five gram of bread. That ____. Q: And sanitary facilities? A: Very bad. They had a hole in the ground and that is where you went and that is uh uh...before you know it there was dysentery and there was sickness and there was uh uh so they had to take us out of there. So after a couple of months they called all the people under twenty years of age and took us into the ____ St. Charles . That is in Marseilles. Marseilles wasn't far from there and they want us to sign up for the French Foreign Legion. So they said you sign up for the Foreign Legion and we give you uniforms and you get food and and you're free. You can go into town. I said well we don't want to fight the Arabs in Africa. We want to fight the Nazis on the mainland. We have no quarrel with the Arabs. We don't want to sign up for the French Foreign Legion. Let us...give us uniform, put us in the French Army, and we will fight for you, but we won't fight the Arabs in Africa. We want to fight the Nazis here. Well, they put us back into the trucks, back into the camp. That went on every other day. They took us into the kitchen to see the fine food you would get and you'll get money and you see the nice uniforms we give you. So I sort of...another guy and myself, we were sort of the spokesmen for the...we were the younger, you know...we ______ the older ones, and I say it doesn't impress us. We still ____. We stay...we stay in the in the brick factory. They had to let us out sometime. We stayed till the _____, but we went and this went on and on. Finally... Q: What were your feelings...uh fear...could you describe it? A: No. I wouldn't say fear. We just accepted things. I don't know. We just...whether it was our age, our young years that uh...no, we weren't really afraid of anything. It's just that 8 things happen and we had to accept it and and and uh...there were times when we cried. There were times when we...when we were a sort of scared a little bit and and but there were other people that would tell us uh uh...I would ask the guy next to me uh what's going to happen to us. He would turn around and says, _______________. That means everything passes, even life. This shall pass too...in German. So I said come on, that doesn't tell me anything. This is something you don't appreciate for ____. You're young. You'll...you'll make it. They have to let us out sometime, so look forward to that. Those guys, they were singing and the guys they told stories and and we had conversations and and we just sort of accepted things, knowing...hoping and knowing that someday we will...it will end. It will stop. We didn't have much time to think about our families anymore, and that's what scared me. We were so busy with things that we never thought about our our families anymore, what happened to them. We had ___ what happened but _____ nothing. We were so busy with our own trying to stay alive in in under those conditions that were _____ us. Well, one day they put us, put people in trucks. Says we were over-crowded there. We're going...they're going to another camp. I said fine. I had nothing to say. And they took my friend. I didn't want to be separated from him. I jumped on the truck too. You had no luggage. You had nothing. You add to the confusion. The French had a knack of confusion, of being disorganized. So we got into the other camp, Monasque , called Monasque. Now here was a...here was a camp that was made out of an old ____ house. They put barbed wire around it and made a camp out of it. A few days later called my name out. I wasn't on the list in in the mill. I wasn't on the list in this camp, and they were investigating what happened to me. And they said to me, well, you have to go back to the other camp. You weren't on the list. I said well, there was nothing much I could do. I said goodbye to my friend _____ and I said somehow we'll meet again. We didn't like to be separated. Well, on the truck they had a few other people from other places that they caught up with. We were going. It was late in the day, almost night. It was cold. And they were going slowly around the bend. I had no idea where I was, and the back was open. They had no guards in the back to watch us. There were two guys. They didn't want to sit in the back where it was cold. The guard was sitting up front next to the driver. Went around the bend. There was an embankment there. I jumped. I got...I caught the opportunity. I went out of the _____, the truck. I jumped down, fell down the embankment, rolled down, and I was laying in the mud. That was it. I had no idea where I was. So I saw the the lights of the truck disappearing on the highway, so I just lay still for a little while. Then I got up and I was hurt, and I walked into the woods. I stayed the whole night, day and night. Early the next morning I heard voices...voices. I was so hungry. I was always so hungry. So I went up there. There was a bunch of guys, Jewish boys, all nationalities. Some of them had guns. Some of them didn't. Some of them had grenades. One of them had this uh knife, so they looked at me. They pointed a gun at me. I says, I am a Jew on the run. So one guy said so what else is new. Have you ever seen a Jew that's not on the run. I said ____ no. He said come. Join us. So they got more guns and he said look, we do some raiding. We do some trouble. All we can do is like a big...put a sting here and a sting there to the Nazis, but that's about as far as we can go. We...we are no match for the armament of the machine guns or for the masses of those people and and their tanks and their and their carriers and whatever they have. We 9 are...but we can give a sting here and there, now and then, so that's what we'll do. Hurt them and run. So that's what we did for a while. Then the word came out there is another group in Langlade . We ought to join them, so we made our way to Langlade, traveling at night, hiding during the day. Q: How many were there of you? A: There must have been thirty, forty. Q: And what were the ages? A: Young and old. From age...from age nine, te n years old up to forty, fifty. Somehow they got out. Somehow they are lucky enough not to get caught. Somehow they were on the run for God knows how long, and somehow they found themselves in that predicament. Nobody asked questions. So when you're hungry, you know you do strange things. We went to little villages, French villages, you know where the bakers put their long breads in baskets outside the store. I was a good runner, and I was a good thief. I learned as I went along. I was hungry. So I said to the guys, get me a few guys, good, the best runners we have. We nonchalantly walked through the little village there. We grab a few breads. They had these long breads, these French long breads. I mean long, not the ones we've got here. About four or five feet long. You grab a few breads and you run like hell, and then we shared and that's what we did. We went in, grabbed the bread. By the time that they had a chance to even see anything, we...we were running like the dickens. We were scared, hungry and scared. We were always hungry and scared. We shared it. Have you ever tried to go into a chicken house and steal a few eggs. Don't ever try it. I did. We went into a farmhouse. We said to the guys...all we could speak was German. We couldn't speak French. We couldn't speak no other language. Nobody talks because if you speak German, the French hate the ____ , the Germans. They're going to kill us. Nobody talked. We get caught, nobody talks. We went into the chicken house to steal some eggs. The chickens carried on. You would not believe the noise they made. The farmer came out. Two farmers came out in fact, with pitch forks. They found us. They would have put those pitch forks right through us because food was always scarce. They didn't...they didn't appreciate anybody stealing their eggs or chickens for that matter. I tell you we never run so fast in our lives over the fields. After a few miles out there, we start eating some of the wine, the grapes that grow in the fields, but the farmers put a chemical on those grapes uh so the flies and things don't uh spoil them. They gave us diarrhea. We got sick from them. It was a bad situation, believe me. We finally made it to Neim , and we went to Langlade two at a time. The others stayed in the woods. Only two at a time traveled. We went to Langlade. There was a small village, small French village, and...a deserted village, nobody there. Empty...just the houses stood there. We went in there. We saw a guard, one our people, one of the crowd, and they took us in. We joined them. They were...they already were better armed, better trained, smarter in in in the things they had to do. After about a week or so we were surrounded...somebody must have gotten wind of it...by the gendarmes, the French gendarmes, and they asked us to give up. Well, 10 we knew we were in the south of France GPE , and you remember well that France GPE was occupied only half and the other one was Vichy controlled, Marshall _____ and _____, you know about that. So at least we knew we were in the south of France GPE . You always had to go south. So we got together, said look, all the young guys, guys like, you know, fifteen, sixteen, twenty, twenty-one, twenty-two...make a run for it over the field. We're going to shoot out at the gendarmes on one side, and you run on the other side. We keep them busy on one side. So we said what's going to happen to you. They said don't worry about us. We are older guys. We expect it...we expected this anytime to happen, so it happen now. Maybe we'll get away too. But we'll fight it out. We...we're going to give them a run for their money. In the meantime you make a run for it, all the young kids, make a run for it, because you're the ones that can run. We can't. And believe me, we got good at running. Sure enough, they started the firing and he says go and we went, and we run over the fields while they keep the gendarmes busy. And the French aren't ____ anyway. We found that out. They they're not too bright. They're cruel. They're barbaric, but they're not bright. And where will you go? You're running through the fields. Where will you go? Naturally...Nice. That's the answer...Nice, toward the Italian border. Nice is toward the Italian border. I end up in Nice and what the French did...they would close up a part of the town. Suddenly three or four trucks with gendarmes would pull up and they would close up off part of the town and and and they would arrest anyone that they can catch that had no papers. You had to show papers. You had to show passports. You had to show money that you are not a vagrant. If you can't explain your existence, to jail you go. Q: Were you by yourself? A: I was by myself. I didn't know these things that they did there. And of course I was caught. So what they do...they put me into this ___________. There is a ______, you know, the ____, a park, an army park, in in in Nice and they had a jail. And they put me in this and took me there. The others they put in other jails that they caught. To...to me no papers, no money, no past, no present, no future. I couldn't speak a word of French and I was hungry. I said what more can happen to me. They put into _________ into a cell by myself...one day, two days, three days, and the jail was right...it was a jail and a little courtyard and the wall and the street. The street was like this. So I was watching. I I I became observant. I could shop . When they put me into that jail, I kept walking slower, I kept sizing up the place. I was looking around to see how many cells there, see what...where the wall is, see what's behind the wall, uh see how the cell closes...I became very, very observant. One day the guard came around. He says, well, he says, one morning, tomorrow you're going to be handed over to the Gestapo. They're going to pick all you Jews up. Well that night I counted the time the guard would make the rounds. It was about ten minutes. I counted...one, two, th ree, four, five minute passed...one minute, two minute, three minute, four minute. About ten minutes. There was a hole in the cell with a little trap door where they put that thing in where you urinate in, and that was closed with a piece of wire. They just place...put a piece of wire through it and wrap it around. They say you couldn't get out. I noticed that. He left. I went up, pulled that thing 11 out and I worked the wire back and forth till it broke. I worked it up. Every time they came by, I stopped. He went. The moment he left I had the door opened. I'm a little skinny thing. I went through. There were five more cells, and they had a bolt that you could open from the outside. I opened all five cells and I looked in. It was a pathetic view. When I tell you it was pitiful...they were loaded with people that they caught here and there, that they arrested, waiting, so I said you know, you got eight minutes to go over the wall. Eight minutes. Now it's seven minutes. Get the hell out. Move. Get over the wall. I got them all out to the courtyard. I said get out, get out. I want to talk to you. They all came out. I said come on. See that wall...make it...make a go for it. Go, go. You had to...if you have to die, die fighting. At least try. They looked at me and they said where are we going to go. Nobody wants us. Nobody's going to help us. Where do you want us to go? Well, I knew I only had a few minutes left and I said look, I don't know about you but I'm going over the wall. I climbed...I went to the wall which was no big deal, but they had this little piece of glass on the wall like like like to cut yourself on, you know, was over every half an inch of glass. I took what shoes I had on, if you could call it shoes. I went in between but I cut myself anyway. Outside the wall was a lamp post and outside there was a car going back and forth. I looked right on top of them. I waited until the ____ walked the other way. I jumped onto the lamp post, went down and start walking immediately. I didn't even stay...I di dn't give him a chance that he would...where this guy come. I just...I never saw him...I just a minute ago...where'd he come from. I walked. He didn't say a word to me. I nodded to him. He smiled at me and I kept on going. The moment I was a few hundred feet away, I started running like the dickens just to get away from that place. Two guys stopped me. One guy said to me if you're on the run, if you're a Jew and you're on the run, three blocks down is a school, a school. They have three basements. Go to the third basement. You can collect your breath for a day or two. Get something to eat. That's what I did. I went into the basement. They took me in. There must have been a hundred guys there, hiding in this school in the basement. And he said to me you can stay maybe two days, maybe three. Then you have to make room for somebody else that needs a piece of bread or that needs to get a night's sleep. So somehow we started to get organized a little. Somebody...you see, it was a big thing when...if you could get a night's sleep without having to worry or if you get a piece of bread without having to sleep with your with your stomach empty. That was a big thing. Or or if somebody gave you piece of blanket and you got warm for a change. That was a big thing. You could let go of fear and of and of of nerves and of and and of being so frightened, and and this so cold all the time and so hungry all the time. You could let go of it a little bit. You could lay down, say ah...I just want to sleep a few hours without having to worry somebody putting a bayonet in my back or a gun to my head. It lasted a few days and said look, I have to make room. I got out. I got caught again. Got no place to go. There was no place to go. This time they took me to a Ausglis Sur Mer . Well Ausglis...naturally in chains with dozens of others. Ausglis Sur Mer. They put us into, into the camp and inside the camp was another tent with barbed wire around it. They threw us in there with maybe uh twenty, thirty kids...all kids. Ten, twelve, fourteen, eighteen, fifteen, thirteen. Look ____. There's tw o boys going to be shot. That's your life there. Make the most of it. Nobody really understood the significance of being shot. 12 Somehow we didn't even...we couldn't comprehend it. Only the fewest mentally could comprehend it. (Pause) Next day came...nothing. Night came...nothing. Day came...nothing. They threw in one bread like this was food was in for the day, and the others break a piece for this one and a piece for that one. Threw some water in, and we waited and one morning they took us out and amongst another three, four hundred they put us onto a unto a cattle boat that was docked nearby and uh the guard du mobile was on each side and we had to go through the line, you know what I'm trying to say, through the line between the guard du mobile, be...between the gendarmes, and as we went through all chained together, they would beat us and kick us and spit at us. Some fell down and we had to drag them with us because they're chained together, to get onto the boat. It was pitiful. It was really pathetic. We had never seen...we had never harmed these people. We never saw these people. Finally the captain of the boat came out and he told them to stop. He said as long as I'm the captain, as long as these people come on my boat, they are not going to be beaten anymore, not by you, not by anyone. They're under my jurisdiction now. Now let these people go. He was a nice man, the captain. We went on the boat and there was a hole on the deck. We went into this hole, and on the top they put some heavy machine gun onto the hole. That was the only way you could get out. And the the boat took off. We had no idea where we were going. Some people, the older guys told us...they looked through the, you know, uh openings in the in the belly of the boat there was little openings that you could look through...say I see the coast of Spain GPE . Now if you see the coast of Spain GPE , they're going to ta ke us to Africa. The coast of Africa is not far from there. So they took us to Africa...not Africa...Algiers. They unloaded us in Algiers in prison. They were waiting for other transports to come. We went to jail. We stayed in jail...must have been maybe six, seven weeks, in solitary confinement. One was not allowed to talk to the other. Let me assure you, the only way we kept our sanity was we knocked on the wall and the other guy knocked back. That was the only outside contact we had. It was a little cell that you could hardly lay down your blankets. No bed, no nothing. We had to lay on stone. We had very little water. We had very little food, and there was just no way that we ever found out what was going to happen to us. We lost...we lost uh uh track of time. We didn't know what day of the week, what month we be...we just didn't any more uh uh mentally uh whether it was day or whether it was night or what. It was dark in there with a little tiny window where they pushed the stuff going and and a little pot to uh uh urinate in and and they didn't clean that out for about a month and uh that was just...there were times when you wished you just gave up. You just...let them kill me...I don't...you didn't care anymore. There was just so much you could absorb. There was just so much abuse and beatings and and and starvation and and thirst and cold and heat that you could ab...that you could take. And this was just enough. You just...many times you thought I wished I don't get up tomorrow. The hell with it. I don't care anymore. You didn't give a damn any more, but finally one day they took us out and they put us...took us to the railroad station and put us into a cattle train where they transported camels and donkeys, and they packed us in there. We had no idea where we're going. And they said well, you'll never make it. By the time we get through with you, there won't be much left of you. Nobody answered. Nobody said anything. There was nothing...there was nothing to be said. And the train started going. 13 Q: Let's let's stop here and then we'll continue on the next tape. A: Right. Right. 14 Tape #2 Q: OK. Can you continue your story? A: Yes. The train started to move and we traveled...we must have traveled about three days. They packed us in like sardines. Some people died standing up. They wouldn't let us out to use the bathroom facilities or anything like that for three days. Q: Was this all men? A: Pardon me? Q: All men? A: Yeah. Of course. All different nationalitie s. Jews from Romania GPE , from Poland GPE , from Czechoslovakia, from Germany GPE , from Austri a, from Hungary GPE ...you name it...they were all there. This was the remnants of people that escaped the persecution of their countries and and escaped through through woods and and swamp land and and borders, and smuggled borders who tried to save themselves uh uh and ended up being being uh in that...being caught and put in that predicament. And uh once a day they threw one bread into the car, to the cattle car, and a few bottles of water and that was for everybody. Q: Did people share? A: Yes. We all kind of...you know, if you're so hungered and so thirsty that you ____ that your tongue swells on, you know, when you're very, very thirsty. I mean very thirsty. Your tongue has a tendency to swell out and you choke on your own tongue. It happened to people. And when you see a drink of water, believe me, you go for it. When you see a piece of bread, you go for it. Well, after about three days in that...it's about two hundred miles into the Sahara, through the desert...the desert as well as swamp land and nothings...I would call it a barren desert...so finally the train stopped and everybody out. They had the Spahice watching us now. The Spahice is an elite Arab police force, called the Spahice. They have sabers and they are on horseback and they are vicious. I mean vicious. They were guarding us. When we were dragging ourselves, we saw a few miles down the road we saw a few tents, tents and barbed wire around it in the middle of nowhere. That was Camp Djelfa . Little did we know...we dragged ourselves through the sand to get into the camp, all chained together. Oh the the French are famous for chaining you up. We were real dangerous people...half-dead, half-starved...I mean halfsick...all kinds of dangerous people...so we dragged ourselves and we dragged the ones that couldn't even walk anymore. We dragged dead ones with us that were chained to us. They were dead. We had to drag them along the sand, and we could hardly ourselves walk. We got into Djelfa very slowly. They were beating us all the way down, with whips, with horse whips. Uh on their horseback they had these long whips. They were whipping us all the way down and if you fell down, they would beat you till you got up, 15 and it hurt. It really hurt. We got there into the camp and we had to stand at attention for the camp commander to come to address us, so we stood there for hours till he finally showed up. He showed up with his German Shepherd and uh he had somebody with him and they pulled out a great big steak and they gave it to the dog and we had to watch how the dog ate that great big steak, and then you heard a speech, and I will never forget if I were to get a thousand years old which will never happen but that's just ____...he says...alright, he says. I am your camp commander. You all came here to die. It's just a matter of time. If you do what you're told, you live a little longer. If not, you die sooner. (Cough) Excuse me. He said but you all came here to die. Sooner or later, you all die. None of you will come out alive. That is my job, to see to that, and I'm very good at my job. Now get into the tents. So they unchained us. He had an adjutant with him. Now the camp commander...let me explain this to you...he was a...the book will tell you that...the guy that wrote that book...he was a sadist and a drunk. And the adjutant...he had an adjutant with him, his right hand...he was al so a sadist. They would put people up for no reason. So we didn't know how bad it was. We found out later that Djelfa was the the worst camp...there were many other camps, ____, _____, Djelfa...there were many other camps there, but Djelfa was the worst concentration camp in all of Africa. The only people that were sent to Djelfa were the ones they rounded up in France GPE and anywheres they could find them, the most dangerous people to France GPE , the most dangerous. That's the ones they sent like a _______. That's the ones they sent to Djelfa, to Africa, to isolate them from the rest of the world and eventually kill them. We found out later. We went into the tents and we waited, and the next day, the next day we had to work. Now they were trying to build a trans-Saharian . They were trying to build a railroad through the desert. The location of the camp was at the foot of the Atlas Mountain MOUNTAIN . That means that at night, the nights were so cold that your teeth would chatter, and the days were so hot that you almost died from the heat. It must have been a hundred and forty degree heat in the day, a hundred and fifty, in the shade. There was no shade. And during the night you were so cold that you couldn't sleep because you had...you had no control of your teeth that would...that that would...you know, uh uh make such noise. We had nothing to cover us ourselves with except the rags that we wore. They gave us one bread a day for a tent, and a cup of coffee in the morning which was black water and a half a liter of soup and a half a liter of soup at night. ____ soup. We were full of lice, bugs, and then you started dysentery. All your blood came out. There were vipers that would bite you... in one hour, you're dead. We were full of scorpions. You know, there are bad things in the sand there. And we were laying in the sand. There were forty, fifty people in a tent made for eight and uh you had only lay on your side, and if one had to turn around, the whole tent had to turn around, and uh turn around...if something...something bite you, there was nothing you could do. You you couldn't do a thing. Even if it was a scorpion or a viper or a snake, there...then you die. That's it, and they were building that front area into the desert and you had to work on it ten hours a day. We had to drag sand from two miles down the road, uh down...two miles down the other road to clear the dunes. And the Spahice would keep whipping you. You must have read about the Egyptians transferring the Jews in slavery. That was much better. It was the same thing...the beatings, the hunger, the thirst, the heat, the cold. (Pause) And the beatings. He would 16 come in the morning and he would have a dozen people striped naked and and whip them in front of the rest of them just because he likes it and he would stand there smiling. The slightest infraction of the rules...they would bury you in the sand, just your head sticking out and the Arabs would urinate on you, and if you moved your head, they would open your skull. They had little tricks like that. The adjutant would come in and just pick anyone he wanted, tie them up and beat them up and whip them. At just...just at will...there was no intelligent reason for any of these things to happen, because people didn't do anything wrong. If you made a fire, the fire...we had no firewood but we used alfa ...that is a desert grass and that burned, so we used to always keep bushels of alfa that we could get a hold of and uh make a fire with it to make coffee. How did we make coffee? We made coffee on dates...uh the the pits of dates. We would brown them...makes good coffee too. The pits of dates...brown them, then with salt we would uh uh ___ and and then we put them in in in hot water. Makes good coffee. The camp commander comes. He sees smoke, and the first thing you would hear, ________________. That means take...you know, we used to eat out of of cans like what you have here, uh vegetable cans that you buy in a supermarket...we had a can. That was our...the extent of our...so he would say get your can and get your things together, whatever you have. You go ______ fifteen days. _________ was a a fortress of the French Foreign Legion, the last outpost of the French Foreign Legion. There was no longer a French Foreign Legion there, but the camp commander lived there and the adjutant lived there. They had their quarters there. He kept his horse there and there was the jail there. To go to Fort _____ for fifteen days meant water and bread, fifteen days. Seventy-five gram of bread a day and a cup of water. They would tie your arms behind your back and hang you on your wrists, like this and they would hang you on your arms, oh for a night, a day, two days, till finally your arms would hang out of the sockets, and then they would come in, strip you down in the middle of the night, strip you naked and put the hose on you. And I took the cold that was at night, we got more cold. The hose didn't do just an easy job on you. If we were lucky...if you were lucky, they didn't turn you around and smack the back of the rifle in in to the small of your back. That was...that was fun. To hear you scream was music to their ears. That was classical music to them. We didn't give them that...no...we didn't do that favor. Not a sound came out of us. They were beating us and beating us and not a sound. We would go...get unconscious, but we would not scream. We became very hard by that time. We became very bitter and very hard and and very unforgiving. We would not forgive ourselves...if we would have put out a scream, we would not forgive ourselves for that, because it gave them satisfaction. We wouldn't do that. After fifteen days...there wasn't much left of you to begin with...after fifteen days, they had to get people to drag you back in, because you could no longer walk. The next day, you had to go back, back to work. Q: Did you have any communication with the other men while you were in this place? A: Oh yeah. We were maybe five, six, or seven on the hook, most of them unconscious from the beatings. Most of them didn't feel like talking. Most of them gave up on life. THey didn't...there was...there was just no communication...what...what was there to say. What 17 could anyone say? It hurts. I'm in pain. I'm hungry. I'm thirsty. I'm tired. I want to get out. I don't deserve this. Why are you doing this? Where is the world? Where is everybody? Where is God? ____ there was no God. There was no world. There was no Pope. There was no priests. There was nobody. And by that time, we came to the realization that we are on our own, not to look out for help. There was not no help coming. We were there and unless we kept ourselves alive, we're going to die there, just like he said. So we made ourselves a promise. We won't do him that favor. We'll survive, and we worked on the trans-Saharian and two years later a dune came...a wind came and the same dune was over the tracks again. The wind just put sand back on the tracks. We went back. We had these sacks we put the sand in and that wasn't good enough. They gave the sack that was in the front and in the back...one sack to carry was not enough yet for them. We had to get a double sack full of sand and sand is heavy. And when you drag yourself through the sand with your feet, when you have no...you're barefooted and the sand is hot and you sink in...in to the sand, it's hard to walk and you carry that weight which it just pushes you down and you have no strength left and you see, you look up and you see this Spahice from the horses looking at you. He's...he's just waiting for a reason to to beat on you. You'll move. You get your last bit of strength together and you'll move. And you hope and look forward for the end of the day when you can go back into the camp and just fall down on your face and get a drink of water. Anyone who could no longer work, was at the end. There were guys...people that just could no longer work, the older people that could no longer absorb that punishment...they just laid down. They would get no ration, no bread ration and no water ration. That was forbode. That was...you know...a strict rule...unless they get up and work, they could get no nourishment. So what we had to do is from the piece of bread that we got, we broke up a little piece into...well, a bowl, whatever you had there, and we collected and we gave it to them...a few pieces, just from a few pieces from the little that we had and gave them a drink of water from mine, a drink of water fr om the other guy's, and we we made another cup full like from the little bit that we could afford to...you know, that we gave. And our...I remember our hands would shake when when we gave that bit of water and we looked at their faces...how they would be looking at us...thank you, thank you, just wet my lips. Then the order came down. Anybody who can't get up, they're going to come around and cut their vein. So they came around and with their knife...they couldn't get up anymore, and cut their vein and bled them to death. And they were...they were looking and says we know. We know what's happening. We know. ____ come to me. We know. And then all the sickness came. By the time...the first thing you did in the morning, you deloused yourself. I got real good at it. We had rags. Mostly we were naked. We had nothing. We deloused ourselves. We had guys with dysentery, the other guys with malaria, with fevers, snake bites, scorpion bites...so what we did, we caught...there are the wandering Bedouins ...they they wander through the desert ____...we would get a female dog, catch a female dog, keep him at the bonfire. The female dog would attract...they always have dogs with them...would come over to the female dog. We would catch them, hit them on the head with a rock. We had butchers in the camp...and butcher the the dogs. Like Mr. Sadowsky that wrote that book, he says we lived mostly on rats, snakes and dog. That is...that's what kept us alive. That's the desert rats. 18 They jump like kangaroos. You've never seen one? Yeah. We learned to live with it...and dog. The first time I ate dog, the guys called me in, said here, eat this. I ate it. It was good. I was hungry, and I said what is it. They said don't tell him. You got use to it. That's what kept us alive in there. They would...they would just beat you up for the hell of it, just to have fun. This camp was about the worst bit of hell on this God's earth. In the rainy season, the tents would fall apart. They would sink into the ground and tip over. And the rain wouldn't stop for six weeks. There was a flood out there, and we would lay in the mud for six weeks. Just lay in the mud. Just no place to go. You couldn't put a tent back up again. It would sink right back in again. And we had no way to take a bath. We didn't take a bath in two years. We didn't get a haircut in two years. We didn't get...we didn't brush our teeth in two years. Just try to stay alive. Some of...who worried about brushing their teeth. It was just about like this, like he said in the book, the worst camp in all of Africa. That's where they put the most dangerous people. I stayed two years in that camp, from 1941 till 1943. How I survived that camp, I will never know. What frightens me the most during those two years, and what I feel guilty about (coughing)...I never thought about my family. I was so busy worrying about myself, I never worried about any of my family. That scared me the most. I was so detached from my family, from life, from the outside world, from life itself, th at I never thought about them. I never thought about my home, until I finally got out and I took my mother's picture, and I I wrote down, _________. I escaped death. We will meet again. I wrote that down on that picture in 1943. On the other side was a little space and I wrote down there, luck doesn't seem to like me very much, but I'll make do. In 1943, in 1942, if you remember well, the Americans had landed in North Africa, and they took over the Hotel _____ which was the biggest hotel in Algiers and they were having a good time already. They were dancing in the streets. They were eating and drinking and having...smoking cigars, and we were still in the camp. When the message got...finally got through...there's a camp over there. Why don't you let these people out? They're dying there everyday. They're on their last leg, on their last bit of breathe. The answer came from the high command...we have our own problems. What are we going to do with these people? Where are we going to put them? How are we going to take care of them? We have our own uh headaches? We'll figure out what to do with them. Leave them there for the time being. (Pause) Now...we stayed there another five months after Algiers was liberated. Five more months in the camp, and died more and more and more in these five months. Finally the British came and got us out. They came down the dunes with their their tanks and half-trucks and they saw a bunch of sick people, dragging them...themselves around, naked and half-naked with long hair, unshaven, dirty, filthy, sick, weak. Who are they? Who are you people? And we had a man in the camp...he was a professor of at the Vienna University...Professor Dr. Pick ...and he spoke oh something like ten languages, and he went forward and he told them who we are. So...but they had their guns pointed at us and all sides surrounded the camp and the machine guns pointed at us, and the big guns pointed at us and ____ with their hand guns pointed at us. They're not...why are you pointing guns at us? We are still dangerous people? Look at us. Look at us. We are the one end of a of a persecuted people that just tried to stay alive, abused and beaten and persecuted and hunted people. Have you no pity? You too. And so they 19 said alright. By tomorrow you'll all be out, and they got the trucks down through the desert and they picked us all up, on stretchers, carrying us. There were still people dying. In the truck there were people dying and we went into Algiers...outside Algiers, a place called ________ which was the 96th General Hospital. Never forgot it. They put us into the hospital. At the hospital they tried to get to us to take blood or give us blood or something. They couldn't get needles through. Our skin was like leather, so dried out and and thick, so hard from lack of food and lack of uh whatever you normally would have had to stay healthy, so they had to take knives and cut through with knives to give us blood. And after a month at the hospital, they said now what are we going to do with you people. They had some intelligence officers talk to us. Are you going...what do you want to do? And they called me in. They said I'll tell you what...you can go anywhere in the free world and you want to go. We'll give you transportation. We take care of it. If you want to go to school, we'll send you to school. If you want to go to England GPE , go to England GPE . If you want to go to Israel, go to Israel, anywhere you want to go, go. And we kept looking behind us because we were looking for the guards to be there. We were so used to looking behind us because everytime in the camp we had to go to the latrine, we had to call out, latrine. And the guards would come with this cocked gun...go. You went to the latrine. They would stay behind you with a cocked gun, wouldn't know whether he's going to shot you in the back or whether you fall into the you know what. And we were so used to this, that we couldn't believe there was no one behind us. So I said to _______...I couldn't speak a word of English...I said tell you what, I said I feel better now. You give me a gun. Now it's my turn to fight. Now this is my time to fight. They had...they had their fun. Now it's my turn to have fun. They looked at me. After all this, you want to fight. I said yeah. I said look, I left home. I lost my family. I lost my country. I lost my home. I lost everything that was dear and close to me, everything I loved. I was hunted like an animal. I was persecuted. I was beaten. I was shot at. I was...I was abused. I was starved. I was beaten half to death. I was worked half to death. I was hanged by my arms until they was out of their sockets. I was hosed down with ice water in the cold of night. I...I...I have seen just about everything. I have absorbed just what everything that this world threw at me. There was no help from anyone. No help came. Though I begged for help, no help came. We licked the walls of the prisons for a little bit of moisture. We knocked on the walls to get the answer to keep our sanity. They have thrown just about everything at us but the kitchen sink. They have taken everything from us including our dignity. What more is there? What more is there left to live for except having a little fun with them now? I said can you understand that, sir? And he just looked at me. He says but you don't speak English. I said I'll learn. But you don't know how to shoot. I'll learn. And after I learn, not just how to use it but to use, that's all I want to do. I want to back to Europe and I want to do my job and I want to see if any of my family has returned or is left alive. Now that is my God...I have earned that right. Don't you think so? He called in the other ____ sergeant. He says give this man a uniform and give him the regular gun. I says no. I want a machine gun. He says why. It kills more people in a shorter period of time. Just give him the plain gun and teach him how to use it. They gave me a uniform. You saw the uniform. I had a little training. I started learning English, the most important words, and I went into battle. That time we battled the German army in North Africa. We 20 _________. I joined the 8th Army, Montgomery's army. They put me into an outfit all foreigners, the whole outfit, and we did the jobs that they couldn't do. When something had to be done, they called us in because we had...by this time we had nothing but hate. We had so much hate in us and so much resentment and so much, and so much bitterness in us by this time that we didn't care about nothing. We didn't care to be shot at. We didn't care whether we get _____. We didn't care whether we fired at them or shoot them. We had no pity. We had...we just had no pity left in us, because there was no pity for us. Nobody gave us an ounce of pity when we needed it most. So they called us in for big jobs and bad jobs and we did it. After about a year or so, a year and a half, they called me out and said we're going to send you to the office in London. That is fine. Sent me there and I went through a course of intelligence, army intelligence. So I said what am I to do? He says well, we don't need you so much shooting up the Germans in Africa. We need you going after war criminals. That's more your speed isn't it? I said is it ever. Is it ever. Sure enough. They educated me in intelligence work. They attached me to the FSS which is the Field Security Service, branch of the Intelligence, un Investigations, and uh they said...they had a big map on the wall and said to me where do you want to go in British occupied Germany GPE . I said well, pick the...pick the city. Well, my home town was occupied by the Russians, so I couldn't go there, so I said well, give me Dusseldorf. They said you've got it. You got the city. Do a good job now. Here are your papers. Report to headquarters. I went to Dusseldorf, report to headquarters, and they said go out to _________, outside Dusseldorf and there's an office waiting for you, and do what you have to do. Get those Nazis. I said you bet. And we traveled in crews of three, a German Jew, a Polish Jew and a Hungarian Jew or Romanian Jew, whatever. Between us we spoke ten languages. By that time I spoke French. By that time I spoke English. By the time, you know, I picked this up from the _____. We're not all that stupid. And we went, we traveled. And did we ever do a job. Did we ever do a job. We started investigating the mayor and the chief of the police. We started way, way on top. Said never mind the little fish. Get the big fish. And we got a lot of them. And then they called us in and said look, you are too tough. They said if we are _____ as the Nazis then we are no better. That's what he said to me, my major said that to me. Then we are no better than they were. We have to stop hating and we have to stop being bitter and learn to forget. I said Major, how do you do that? You have to _____. How do you forget? He said, I know it's hard. It'll take a long, long time. He said so what. What are you going to do? We used to do bad things. I mean, you know, we used to get them and used to blow their brains out and tried...they tried to escape. We used to put them and they beg us to...I I remember in Holland we caught one. He was hiding in a baker's oven. (cough) We caught him and we gave him a choice...you stay there. We put the oven on. If you come out, I'll blow your brains out. He said that's no choice. I said that's more the choice you gave our people. Our people didn't even have a choice. You threw two hundred thousand children in open fires then burned them alive. They had no choice. You gassed _____ our people. Did you give them a choice? They had no...not even a choice how to die, just make them die. You murdered them in cold blood. Did you give them a choice? No. We are good to you. We let you say your prayers. He came out. I said we give you five minutes. Talk to your God. (cough) That's more than you give our people. And that's the kind of things we did. I 21 mean we pulled in very few, and they found out. And they said...the Major said look, I think you've had enough. I think...I think this is enough for you...no more. Go home. I said where's that? Go to England GPE . Go to school. You're being discharged. After four years and three months, I lost my usefulness. He said no, we could use you. You speak German, but you're too brutal. I says brutality is something that they taught me. I wasn't born with that. OK. I go. Perhaps you're right. And just gave me my discharge and uh I went to England GPE and after a year I couldn't just found myself...I had nothing and I went to the American Counselor. I said I'd like to go to America GPE , start a new life. So the American Ambassador just looked at me. Come back tomorrow. The next day he had all my life in front of him. He told me ever ything that...that he could possibly know, from the day I left home. He said is that you. I said to the teeth. And he said, alright, he says. Here is your papers. Take your papers. Go to America GPE . Good luck. Q: Do you know about your family now? A: Never found out. I found out one thing. During the occupation with Intelligence, I found out that my brother Paul was gassed in Auschwitz CAMP , that my mother and three sisters lost their lives in Wiegar concentration camp in the ovens. Some say they hang them...some say they put them in ovens, and that's all I ever found out. Well, I say, it's _____ I I did things and I tried to put that in the back of my mind, but let me tell you...to describe the cruelty and the barbaric cruelty of the...of the French people, of the French gendarmes and the French guard du mobiles, believe me when I tell you, when I tell you that the French, they're eager to please their German masters. They were...they were brutal. They treated us poorly. When they made their rounds, they beat up children. They beat up women. They beat...they were thrown onto the trucks like garbage and and and and it was just so sad. I was on top of the truck and I helped them pull them in. They beat up little boys and girls and the mothers and that they arrested in the streets. The cruelty displayed by these people, by the gendarmes and the guard du mobile is next to none. I mean they really pleased their German masters, let me tell you, and very little was said about that, the brutality of these people. And I went back to my home town once hoping to find someone...nothing. So I retraced my steps and I put all these years in the back of my head, but let me tell you something...this comes straight from my inside...there isn't a day in the last fifty years, a day of my life...there is not one day, even in America GPE , that I don't think back. Not just my family. Look what happened to me, and there is more and more details are coming out, more and more heroics is coming out and more and more suffering is coming out, and I ask myself...I keep asking myself, how did I ever survive this brutality, this sickness, these sick people? How...how could I ever absorb this punishment, so much for so long and how I will ever understand the indifference of the world? They say...they say that the biggest tragedy (clearing throat) of the 20th century was the Holocaust. I don't say that. I say the biggest tragedy of the 20th century was the indifference of the world. That is what I say, and nobody came to our aid. Nobody helped us. Nobody raised a word of protest, not the Pope, not the kings, not the presidents, not governments. In this whole world...Hitler held a speech in 1938. I heard the speech. He said take the Jews as he spoke to the world...or we will kill them. That's what he said. Not 22 only did they not help us, not only did they not give us a hand, but they closed the borders. They rejected us. They didn't want us. They pushed us back. In other words, they said to Adolph Hitler, kill them. We don't want them. We don't want to help them. We have other things to do. Do with them what you want. When they asked the Allies to bomb the tracks, the railroad tracks that go to to Auschwitz CAMP , the answer came back...we have no planes to spare. When they said they're throwing two hundred thousand children in the open fires and burn them alive, the answer came back...it is an internal matter. When the ship with eight hundred people, the St. Louis SHIP I believe it was, when that uh got in into the United States GPE to Cuba, from the United States GPE , they would not allow them to land. They were eight hundred people, and they would not allow them to come into the country and they went back to die. That is the biggest tragedy of the 20th century, and that is something I will never forgive them for. Not anyone, any of the countries, any of the rest of the world. There are people that...in Mexico there was an earthquake. Millions of people were ____...help, help, help, help, help everybody. When the Shiites had a problem, millions of dollars, planes...they don't know what to do for them. Q: Uh you you...can we just talk a little bit...you came to the United States GPE ...and what did you do? A: Yeah. I came to the United States GPE with three cents, and an old army uniform, fatigues...that's all I had between me and nothing. So I stood on the dock and I went to a bus that was standing there, and I said to the bus driver, you go? And he said to Newark. I said what's Newark? He said outside New York. I said I go there too. I can't stay here. And he says what's...where were you? He said do you have money? I said yeah. I pulled out three cents, English pennies. ____ get in the back. Took me to Newark. I said where do you want me to get off. They had a lot of gas stations. Let me off where there are a lot of gas stations. I walked into this...I walked into the second station, two Jewish boys. I said...I said I need a job. They said to me, so do a lot of people. I said no, but I I need a job now and you've got to give me a job because I can do anything better than you in fact. He looked at me. He said you have a loud...you have a big mouth. I said but I can back it up. He said where are you from. I told him off the boat. You have money? Yeah. Three cents. The bus driver never took the money. He looked at me. He says alright, I give you a job. Thirty dollars a week, six days. I says fine. He gave me something to eat. He gave me...got me a room, and I worked with him in the gas station...washing cars, greasing cars, towing cars. And after a while I said to myself, there's nothing I want to do with my life. And I saw an ad in the paper...school...uh ________. I wanted to do that. I mean that takes ten years to learn, thousands of dollars in tuition. I can't afford that...no money. I saw another ad...watchmaking. Well, doesn't sound bad. I have good hands. I have a good mind. I went up there and I saw the blueprints on the wall. I was running out. The instructor grabbed me by the neck and pulled me back. Stay. I stayed. The course was something like I believe twenty-four months. Seventy-five dollars a month. I stayed. I got myself a job with ______ Cable and Wire in _____, New Jersey...a factory, loading trucks on a platform, from seven to four. At five o'clock I was in school, five days a week. I couldn't pay seventy-five dollars a month. I said tell you what...give me half the 23 tuition off and I'll wash your floors for you. He says you've got a deal. I paid him whenever I could. There were months I couldn't pay him. So I cleaned the place. I took care of the place. I took care of the tools and the land. After twelve months he called me into his office. He says, here's your diploma. There's nothing more we can teach you. I said I have another twelve months to go. He says no, you go out. Go out in the field. You've learned all we can teach you. I went out in the field and I learned and I worked and I learned and I worked and I got better and better and better and I spent nights studying books, uh mathematics, algebra, ratios. I practiced uh on the lathe. I made parts by hand and I wanted to know why this is round and that is square and this is thick and this is thin and why and how...why couldn't it be improved upon and how many ways are there to do a repair job and I learned from masters. I knocked on their doors. I said teach me. Teach me all you know. They did. I became good. I've been doing this for forty-five years. I've had my own business for the last twenty-five years. Q: Before we finish is there anything that you want to add? A: No. I think I covered it. (Sigh) It is just...I tell you before I finish up...this is a hurt that will stay with you the rest of your life. This is...this is a feeling of guilt that will stay with you the rest of your life, the guilt being that you are left alive and they didn't make it and you feel guilty about that. And that will all stay with you and you will suffer the rest of your life with all this. This is not something that will go away ever, and you look at the world and you say, why didn't you help me? You could have done something. You helped other people. Why not us? Q: You had mentioned uh something about heroics...some heroic stories. Is there one particular one you'd like to tell? A: My God...there are so many. They are _____. Yes I do. Twice I spent time in Fort _____ hanging on the hook for stupid reasons. The third time would have killed me. I had made a fire. We had lookouts. They didn't watch out good. They caught me and I was supposed to go another fifteen days at Fort ______. They knew if I would go another fifteen days it would be the end of me. There was nothing left of me after two to start with. Before I knew it, all the remain camp guys, they assembled in front of the gate and they had me inside and they said no, he's not going. It came out sudden. He is not going to Fort ____. Neither is anybody else. We've had enough. Nobody is going out of this camp. Nobody is going to be hanging up their arms. Nobody is going to get beat up again. Nobody's going to die anymore. We've had it. The guards could do nothing. We were a thousand, two thousand guys there. So they told me you stay where you are. You're not going anyplace. Everybody picked up rocks. That's it. So they called in the Spahice. We saw them arrive with their horses. He called them in. They came through the gate and that was the only uprising we ever had...could do in the camp. There was two-fold...the reason was twofold. No...nobody ever again would go to Fort _____, and maybe this will go on to the outside world to let people know what's goi ng on here, make people find us, make people investigate. Perhaps somehow, somebody will bring it into the outside world that other 24 people died. They came with their sabers. We threw rocks at them. We trampled them to death. We took their sabers and we fought. It was a bloody battle in the camp, but there wasn't much we had to fight with. And I didn't go to Fort _____ that day. It was spared me, but people actually gave their lives to save mine. I think that's heroic. Some how or other incidents...guys that tried to commit suicide uh and went into the barbed...we weren't allowed to go within ten feet of the barbed wire. They would shoot you. And guys started running into the barbed wire. We pulled them back and they were shooting at us and yet still they pulled them in, in spite of the bullets. You did so many things...there was so much going on with so little we we could do about these situations, but yet every little thing was a heroic act. Can you understand that? Everything was a heroic act and and and finally we got out and we said well...Max came to me and says I told you you'll make it, because many days we stood on the barbed wires and says Max, we'll never get out of here. Max was ten years older. Max, we'll never make it. It's already two years. We'll never get out of here, and Max said, you'll see...tomorrow. You'll see...tomorrow. That little...you'll see tomorrow...gave me hope for another twenty-four hours. Can you understand that? Although I didn't...I didn't believe him...thought he was lying and I knew he was lying...but he said it anyway and and I could make myself believe it. These are the little things that kept us going, little things, little supports, supporting each other. Can you understand that? We had so little to support ourselves with and so little hope. How I survived I never know. I tell you to this day it's a miracle how I got out of it, how I survived all these nonsense, all these cruelties, all these beatings, all this brutality. I don't know how, to this day, but I'm here to talk about it. Q: Well, thank you very much for telling about it. A: It's not a hell of a lot, but what can I say. Q: Thank you. A: Thank you for having me to tell my story. TECHNICAL CONVERSATION A: It's a sad story, but nevertheless, it's my story. Q: It's your story and you expressed it... A: It's my story. How can you put years and years of these things...how can you tell about the nights, the lonely nights and the lonely, the hard days? How...how can you put that into words. TECHNICAL CONVERSATION 25 END OF INTERVIEW

This result is not perfect, but again, that’s not the point here. Here we are less interested in catching all entities as much as not catching any false positives. At a quick glance, we have achieved that. Now that were are tentatively happy, wan save our pipeline to disk, but first, let’s add some metadata.

16.13. Saving a spaCy Model

nlp.meta["name"] = "ushmm_rules"
nlp.meta["version"] = '0.0.10'
nlp.meta["author"] = "W.J.B. Mattingly"
nlp.meta["author_email"] = "wjbmattingly@gmail.com"
nlp.meta["description"] = "This is a pipeline of heuristics to help identify essential entities for research into the Holocaust."
nlp.meta['url'] = "www.wjbmattingly.com"
nlp.to_disk("models/rules_pipeline")

If we are particularly happy with our results, we can even save the file to disk. It is important to note that we need to copy and paste all of our code into a python script that we can inject into the model. I’ll cover this in greater detail in the next notebook as we start working on an ML model.

!spacy package models/rules_pipeline models --code data/components.py --force