Commit d887f88c authored by Fize Jacques's avatar Fize Jacques
Browse files

Init Commit. Copy from the bitbucket

parents
# Eval Campaign - Document Matching - 2017
## Database
### Initialize Database
First, create a user.csv, which contains users data. It should have this syntax:
name;email;password;level
Admin;admin@gmail.com;admin;1
Then, initialize the user database, run the next script:
$ python3 database/database.py
#### Admin user:
* email: admin@gmail.com
* password: admin
### Add User
Sign in as an Administrator, then go to : {server.name}/signup
## Run the server
$ python3 server.py
# Credits
Jacques Fize, jacques.fize[at]cirad[dot]fr, 2017
title;path;part_of
EPI10;corpus/epi10.json;
MSF10;corpus/msf10.json;
This diff is collapsed.
{
"title": "EPI10",
"data": [
[
"Highly Pathogenic Avian Influenza Virus Detected in Russia",
"Poultry News Highly Pathogenic Avian Influenza Virus Detected in Russia01 October 2014 
\n
\nRUSSIA - There has been two outbreaks of H5N1 Highly Pathogenic Avian Influenza in poultry reported in the Altai Kray region in Siberia in central Russia. The outbreak has affected 326 domestic ducks, geese and chickens which were located in small villages. 
\nAs a result 322 birds died and the remaining animals were destroyed. 
\nThe source of the outbreak is thought to be contact with wild species, Probably, hunted ducks and geese trophies had been placed in backyards where mortality occurred later in domestic birds.
\n
\n
\nFurther Reading
\nYou can visit the Avian Flu page by clicking here.
\n
\nThePoultrySite News Desk "
],
[
"Newcastle Disease Reported in Romania",
"Poultry News Newcastle Disease Reported in Romania 14 November 2014 
\n
\nROMANIA - There has been a new outbreak of Newcastle Disease reported and confirmed in the Ialomita region of Romania. The outbreak which started on the 1 November 2014, infected a total of 63,294 four-week-old broilers. 
\nAs a result, 50,807 birds died and the remaining 12,487 were destroyed. 
\nThe source of the outbreak is still unknown. 
\n
\n
\nFurther Reading
\nFind out more information on Newcastle Disease by clicking here.
\n
\nThePoultrySite News Desk "
],
[
"South Korea to disinfect farms as foot-and-mouth, bird flu spread",
"\nSEOUL South Korea will disinfect farms around the country over the new year and limit the transport of animals, stepping up its effort to contain an outbreak of foot-and-mouth disease that has spread close to the capital as well as bird flu.The foot-and-mouth among hogs had been limited to the center of the country until this week but a case has now been confirmed at a farm just 50 km (30 miles) from Seoul in the north of the country.The outbreak began five months ago, raising fears about food safety. Nearly 23,000 hogs have been destroyed to contain the outbreak, according to a statement from the agriculture ministry on Tuesday.All the cases have involved a type of the disease that animals are inoculated against in South Korea. The ministry said additional vaccine would be given to farms in affected areas and all farms in the country would be disinfected on Dec. 31 and Jan. 1.\n \n The late July outbreak was the first in more than three years but it came as authorities struggled to contain a problem with bird flu, which has pushed up the price of pork as consumers bought alternative meat.\n \n In its statement, the ministry said more than 40 poultry farms, mostly in southeastern provinces, plus a market near the capital had been affected since late September and 526,000 birds had been slaughtered. No humans have been infected.The livestock disease problems are a big factor behind a jump in pork imports, mainly from the United States and Germany. These rose 21 percent to 328,241 tonnes in the first 11 months of the year from the same period in 2013, South Korean customs data showed.\n \n (Reporting by Meeyoung Cho; Editing by Alan Raybould)"
],
[
"South Korea increases efforts to contain foot-and-mouth outbreak",
"\nSEOUL South Korea is intensifying efforts to contain an outbreak of foot-and-mouth disease by inoculating all hogs in affected areas, the agriculture ministry said in a statement on Thursday.South Korea earlier this month confirmed a fresh case of foot-and-mouth disease at a hog farm, five months after an outbreak elsewhere added to fears about food safety.So far nine cases have been confirmed at Chungcheong province, about 100 kms (61 miles) southeast of the capital, Seoul, the ministry statement said.All outbreaks were the type that animals are inoculated against in South Korea, said the ministry.\n \n Over 13,600 hogs have been slaughtered at the affected farms and Seoul has heightened its foot-and-mouth watch level to alert from caution, it added. \n \n The late July outbreak was the first in more than three years but it came as authorities struggled to contain a continuing problem with bird flu, which has pushed up the price of pork as consumers bought alternative meat. Livestock disease problems are a big factor behind a jump in pork imports, mainly from the United States and Germany. These rose 21 percent to 328,241 tonnes in the first 11 months of this year from the same period last year, South Korean customs data showed.\n \n (Reporting by Meeyoung Cho; Editing by Michael Perry)"
],
[
"Ten Farms in Bauchi State, Nigeria, Hit by Avian Flu",
"Poultry News Ten Farms in Bauchi State, Nigeria, Hit by Avian Flu06 March 2015 NIGERIA - Ten farms in one state, Bauchi, in central Nigeria, are currently battling highly pathogenic avian influenza.Ten poultry farms in Bauchi state were confirmed to have been affected with avian influenza, according to the Ministry of Animal Resources and Nomadic Resettlement.
\nLeadership reports that the Commissioner overseeing the Ministry, Alhaji Umar Sale Giade, disclosed the outbreak of the birds’ disease popularly known as Bird Flu during a visit to Toro local government area.
\nAlhaji Umar Sale revealed that so far, three local government areas of Katagum, Bauchi and Toro local government have been infected by the disease.
\nSeven poultry farms were reportedly affected with the disease in Toro local government area, stressing that 558 birds of various species were affected in the seven farms.
\nHe explained that out of the seven affected farms, 9,208 chickens were killed, saying Toro local government area has the highest population of poultry farms in the state.
\nThe Care-taker Committee Chairman of the Local Government, Alhaji Adamu Umar Danyaro Geji, had earlier while speaking reiterated his council’s determination to partner with Ministry of Animal Resources and Nomadic Resettlement to fight the disease in the area.
\nRepresented by the Head of Administration of the local government, Alhaji Usman Muhammed Mashema, the council chairman commended the state government for its effort towards enlightening poultry farmers on the danger of avian influenza.
\n
\nFurther Reading
\nYou can visit the Avian Flu page by clicking here.
\n
\n
\nThePoultrySite News Desk "
],
[
"Foot-and-mouth disease found in east China",
"
\n
\nBEIJING, Jan. 6 (Xinhua) -- Live pigs in east China's Anhui Province have been found to be infected with foot-and-mouth disease, the Ministry of Agriculture announced Tuesday.
\nThe virus was detected in 556 live pigs in several farms in Maanshan City, 314 of which were dead on Jan. 2, and test results confirmed on Tuesday that the pigs were carrying the A-type virus, said the ministry.
\nAll infected pigs and another 56 raised with them have been destroyed, and the epidemic contained, the ministry said. 
\n
\n"
],
[
"Bird Flu Returns to India",
"Poultry News Bird Flu Returns to India15 April 2015 INDIA - An outbreak of H5N1 highly pathogenic avian influenza has been confirmed in the state of Telangana, which is in the south of the country.Scrambling to contain the spread of the highly pathogenic H5N1 bird flu virus, the Telangana government on 14 April ordered the immediate culling of 145,000 chickens on poultry farms located in a one-kilometre radius of the infected farm in Thorrur village of Hayathnagar mandal in Ranga Reddy district.
\nThis is the first ever outbreak of the disease reported in the states of Telangana and Andhra Pradesh, according to Times of India on 15 April.
\nSimultaneously, a ban on trading of all poultry products from farms in a 10-kilometre radius 'surveillance zone' was put in place.
\nThe outbreak of the disease had an immediate impact with the Telangana Poultry Breeders Association staring at a loss of over 300 million rupees (INR), especially after Andhra Pradesh banned entry of all poultry products into its territory.
\nAuthorities in Hyderabad said the outbreak came to light after several birds started mysteriously dying in Srinivasa Reddy Poultry Farm owned by V. Balakrishna Reddy in Thorrur village from 8 April, prompting them to send bird samples to a testing laboratory in Bhopal.
\n\"The tests confirmed that it was the H5N1 virus and we advised the animal husbandry department in Telangana and the Centre to take all appropriate measures,\" D.D. Kulkarni, joint director, High-Security Animal Disease Laboratory, told Times of India from Bhopal.
\nWithin hours, dozens of animal husbandry workers wearing protective gear, began the culling operation in the infected farm and put the birds and eggs in huge gunny sacks and buried them in deep pits.
\nDr D. Venkateswarulu, director of animal husbandry department, told reporters in the afternoon that the culling operation in farms located in one-kilometre radius would be completed in the next 48 hours. This will be followed by sanitisation and disinfection operations.
\nA Central Rapid Response team comprising Dr S.K. Jain and Dr Pranay verma, both avian influenza specialists, is arriving in Hyderabad to oversee the culling.
\nMeanwhile, the department staff have begun collecting serum and dropping samples of the birds from farms in the 10-km radius to be tested for the virus. If any of the tests come positive, then the government will widen the culling operations. Containment in the surveillance zone will be in place for a week or 10 days until all tests are completed.
\nDr Venkateswarulu said they were collecting information on egg sales over the past 20 days from Balakrishna Reddy's farm, to find out where they were transported. If it is found that the stock is yet to be sold, they will order destruction of such eggs. If the eggs were sold to other states, then the department will immediately notify the states and provide them information so that the eggs can be destroyed.
\nThe official said that chicken or eggs procured from farms outside of the surveillance zone can be consumed and there was no cause for panic. If the chicken is well cooked or the egg boiled, then there is no danger of contracting the virus by humans. If these are cooked at 70°C and above for 20 minutes, the virus will die, Dr Venkateswarulu said.
\nWhile weather does play a role in the spread of flu, Dr Venkateswarulu said that the most accepted theory is that the disease is spread by migratory birds which is contracted either by the chicken directly or through an intermediary carrier bird species.
\nIn 2006, the government in Andhra Pradesh carried out some preventive culling but the enormity of the task this time around has everyone anxious. In Hayathnagar, authorities supplied 2,500 tamiflu tablets to health workers. Health department officials also began a massive house-to-house surveillance with 300 Asha, ANMs and multi-purpose health workers, said Dr G Srinivas Rao, joint director, Epidemic Cell, Telangana.
\n\"We are not leaving anything to chance and will take up all efforts to stop the virus from spreading. The swine flu isolation wards will be used in case there is any need,\" he told Times of India.
\nSince 2003, there have been 400 human deaths globally from the H5N1 strain and hundreds of confirmed cases of infection. In India, the virus was first detected in Jalgaon, Maharashtra, in 2006 and ever since reports of fresh outbreaks have been sporadically reported from all corners of the country. The last outbreak occurred in Kerala in January 2015.
\n
\nFurther Reading
\nYou can visit the Avian Flu page by clicking here.
\n
\n
\nThePoultrySite News Desk "
],
[
"Russia prohibits import of live pigs and pork products from Latvia",
"Federal Service for Veterinary and Phytosanitary Surveillance service Rosselkhoznadzor imposed a ban on the import of live pigs and pork products from Latvia, the service announcement indicates.
\n \t
\n\t
\n\tThe ban came into force yesterday
\ndue to classicalswine fever, reports ITAR-TASS/LETA/Nozare.lv.
\n
\n 
\n
\n\"The European Union is an
\nunfriendly region in case of classical swine fever,\" deputy to the Rosselkhoznadzor director, Alexei Alexeyenko said.
\n
\n 
\n
\nAlexeyenko indicated that the lift
\nof the ban will depend on the changes in situation in Latvia.
\n
\n 
\n
\nRussia imposed a ban on imports of
\npork from Lithuania and Poland on January 30, 2014 over a threat of African
\nswine fever infection. A previous ban like this was imposed by Belarus.
\n
\n 
\n
\nIn January, when African swine
\nfever was discovered in southern Lithuania, Russia imposed a ban on imports of
\npork and live pigs from all countries of the European Union.
\n
\n 
\n
\nAccording to Director at Food and
\nVeterinary Service, Maris Balodis,
\nconsidering the complicated relations between Russia and Ukraine, talks of
\nrestoring the export to Russia might last for years, regardless of Latvia's and
\nother European countries' efforts to achieve normalization of the situation.
\n
\n"
],
[
"China imposes new curbs on Norway salmon",
"\nChina has imposed new restrictions on imports of Norwegian salmon citing fears of a fish virus, Norwegian authorities said on Monday.\n\tAs of Monday, Beijing has banned imports of whole salmon from three Norwegian regions amid concerns they carry infectious salmon anaemia (ISA). That ban represents one-fifth of Norway's salmon exports to China.\n\t \n\tFor salmon from other regions of the Scandinavian country, Chinese authorities have imposed stricter controls aimed at detecting ISA and pancreas disease.\n\t \n\tNorway's food safety agency was informed by letter of the decision, which it judged to be baseless but which strikes yet another blow to the country's star export product.\n\t \n\tSales of Norwegian salmon to China plummeted after the 2010 Nobel Peace Prize was awarded to Chinese dissident Liu Xiaobo.\n\t \n\tNorway's food safety agency stressed meanwhile that Norwegian salmon was safe.\n\t \n\t\"We believe there is no risk that Chinese salmon will be contaminated by the ISA virus because fish products from Norway go directly to consumption,\" it said in a statement last week. \"The ISA virus is not harmful to humans,\" it added.\n\t \n\tThe awarding of the Nobel Prize to Liu plunged the two countries' bilateral ties into a deep-freeze, and led to a tightening of veterinary controls on Norwegian salmon.\n\t \n\tIts share of the Chinese market dropped from 90 to less than 30 percent. The new restrictions come on the heels of an import ban announced by Russia -- another important market for Norway -- in response to western sanctions against Moscow over the Ukraine crisis.\n For more news from Norway, join us on\n Facebook\n and Twitter.\n "
],
[
"Poland's pork meat exports return to growth path",
"\n Poland’s exports of processed pork meat have returned to growth in the first four months of 2015, with an increase of 6.5% to 128,600 tonnes (t), according to the Polish Ministry of Agriculture.\n \n Last year, exports dropped as a result of Poland’s trade disputes with Russia and the spread of the African swine fever (ASF) in some parts of the country, but this year, local meat industry players improved their sales to foreign markets. In the first four months of this year, the reported growth was mostly driven by increased sales to the EU markets, which rose by 27% to some 108,200t.This said, restrictions were placed on Polish unprocessed pork meat by Russia and some other Asian countries following last year’s outbreak of ASF. As a result, Poland’s exports to non-EU markets were down by about 42% to a total of 20,400t in the January-April 2015 period.Italy and GermanyItaly and Germany both increased their imports of Polish unprocessed pork meat, up 25% each, while exports to the UK nearly doubled to 4,900t in the first four months of 2015.The US imported 7,400t from Poland, which represented an increase of 69%, and allowed the country to become the largest importer of Polish unprocessed pork meat outside of the EU. In the first four months of 2015, some 4,900t of unprocessed pork meat were exported from Poland to the UK, which doubled the result from the same period a year earlier, as indicated by figures from the ministry.  On a related note, commenting on the latest data released by the ministry, analysts from local bank BGZ BNP Paribas said that, “taking into consideration the changes in exports and imports [of pork meat], Poland remains a net importer of unprocessed pork meat. The country’s trade deficit increased in the analysed period by 1,800t to 85,000t.” Unprocessed pork exportsWhile the country managed to raise its exports of unprocessed pork meat from January to April 2015, its imports also rose by 4.8% to some 213,500t.Some of the major exporters of unprocessed pork meat to Poland from within the EU include Belgium, Germany and Denmark, respectively. The three countries combined are responsible for about 72% of the country’s imports of unprocessed pork meat. Meanwhile, local analysts suggest that Poland’s pork meat consumption will further increase this year. In 2014, the country’s pork meat consumption rose by 8.4% to 38.5kg per capita, compared with some 35.5kg a year earlier.This year, an average Pole is expected to consume 39.5kg of pork meat, according to the latest estimates from the country’s Institute of Agricultural and Food Economics (IERiGZ). \n \n "
]
]
}
\ No newline at end of file
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
# -*- coding: utf-8 -*-
import pandas as pd
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column,Integer,String, Boolean, ForeignKey, Text
from sqlalchemy.orm import sessionmaker
import bcrypt
engine = create_engine('sqlite:////Users/jacquesfize/db/test.db', echo=True)
Base = declarative_base()
class Corpus(Base):
__tablename__="corpora"
id = Column(Integer, primary_key=True)
title = Column(String(80), unique=True)
file_path = Column(String(120), unique=True)
part_of = Column(Integer, ForeignKey('corpora.id'))
def __init__(self, title,path,part_of=None):
self.title=title
self.file_path=path
self.part_of=part_of
class User(Base):
__tablename__="users"
id = Column(Integer, primary_key=True)
name = Column(String(80), unique=True)
email = Column(String(120), unique=True)
password = Column(String(120))
level = Column(Integer)
def __init__(self, name, email,password,level=1,sign_up=True):
self.name=name
self.email=email
self.password= password
if sign_up:self.password=bcrypt.hashpw(password, bcrypt.gensalt(14))
self.level=level
def check_password(self,password):
return self.password == bcrypt.hashpw(password.encode(), self.password)
def is_authenticated(self):
return True
def is_active(self):
return True
def is_anonymous(self):
return False
def get_id(self):
return str(self.id)
class Annotation(Base):
__tablename__="annotations"
id = Column(Integer, primary_key=True)
corpus_id = Column(Integer,ForeignKey('corpora.id'))
user_id = Column(Integer,ForeignKey('users.id'))
data=Column(Text)
finished = Column(Boolean)
def __init__(self, corpus_id,user_id,data,finished=False):
self.corpus_id=corpus_id
self.user_id=user_id
self.finished=finished
self.data=data
###################################################
# Database Population functions
###################################################
def add_users(session,data):
for d in range(len(data)):
line=data.iloc[[d]].values[0]
user=User(line[0],line[1],line[2].encode(),line[3])
session.add(user)
session.commit()
def add_corpora(session,data):
for d in range(len(data)):
line=data.iloc[[d]].values[0]
user=Corpus(line[0],line[1],line[2])
session.add(user)
session.commit()
if __name__ == '__main__':
# Create tables (delete if exists)
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
# Initialize session
Session = sessionmaker(bind=engine)
session = Session()
# Load data
user_input=pd.read_csv("user.csv",sep=";")
corpora_input=pd.read_csv("corpus.csv",sep=";")
# Populate the database
add_users(session,user_input)
add_corpora(session,corpora_input)
# -*- coding: utf-8 -*-
import argparse,json,glob,re,os
import numpy as np
import argparse
parser = argparse.ArgumentParser()
parser.add_argument("input", help="Input filename")
parser.add_argument("sub_corpus_size", help="sub corpus size")
parser.add_argument("output", help="Output filename")
args = parser.parse_args()
corpus=json.load(open("corpus/epidemio.json"))
data=corpus["data"]
sub_corpus_size=10
size=len(data)//sub_corpus_size
sub_groups = np.array_split(np.array(data),size)
sub_groups = [len(a) for a in sub_groups]
groups,t=[],0
for i in range(len(data)):
for j in range(len(data)):
if j > i:
groups.append([i,j])
nb_of_docs=(sub_corpus_size*(sub_corpus_size-1))/2
size=len(groups)//nb_of_docs
sub_groups = np.array_split(np.array(groups),size)
np.unique(sub_groups[0].flatten())
len(sub_groups)
os.makedirs("generated_corpus/")
[uwsgi]
module = wsgi:app
master = true
processes = 5
http-socket = :5000
chmod-socket = 660
vacuum = true
die-on-term = true
digraph {
rankdir=LR;
subgraph cluster_01 {
label = "Legend";
node [shape=plaintext];
key [label=<<table border="0" cellpadding="2" cellspacing="0" cellborder="0">
<tr><td align="right" port="i1">Inclusion</td></tr>
<tr><td align="right" port="i2">Adjacent</td></tr>
<tr><td align="right" port="i3">Migration Flow</td></tr>
</table>>]
key2 [label=<<table border="0" cellpadding="2" cellspacing="0" cellborder="0">
<tr><td port="i1">&nbsp;</td></tr>
<tr><td port="i2">&nbsp;</td></tr>
<tr><td port="i3">&nbsp;</td></tr>
</table>>]
key:i1:e -> key2:i1:w [color=red]
key:i2:e -> key2:i2:w [color=blue]
key:i3:e -> key2:i3:w [color=green]
}
g [label="Greece"];
m [label="Macedonia"];
eu [label="Europe"];
i [label="Idomeni"];
s [label="Syria"];
h [label="Homs"];
a [label="Aleppo"];
p [label="Polykasto"];
p -> g[color=red,penwidth=2.0];
h -> s[color=red,penwidth=2.0];
a -> s[color=red,penwidth=2.0];
i -> g[color=red,penwidth=2.0];
s -> g [color=green,penwidth=2.0];
m -> g ->m [color=green,penwidth=2.0];
m -> g ->m [color=blue,penwidth=2.0];
g -> i [color=green,penwidth=2.0];
}
digraph {
rankdir=LR;
subgraph cluster_01 {
label = "Legend";
node [shape=plaintext];
key [label=<<table border="0" cellpadding="2" cellspacing="0" cellborder="0">
<tr><td align="right" port="i1">Inclusion</td></tr>
<tr><td align="right" port="i2">Adjacent</td></tr>
<tr><td align="right" port="i3">Migration Flow</td></tr>
</table>>]
key2 [label=<<table border="0" cellpadding="2" cellspacing="0" cellborder="0">
<tr><td port="i1">&nbsp;</td></tr>
<tr><td port="i2">&nbsp;</td></tr>
<tr><td port="i3">&nbsp;</td></tr>
</table>>]
key:i1:e -> key2:i1:w [color=red]
key:i2:e -> key2:i2:w [color=blue]
key:i3:e -> key2:i3:w [color=green]
}
g [label="Greece"];
m [label="Macedonia"];
eu [label="Europe"];
sy [label="Syria"];
au [label="Austria"];
tu [label="Turkey"];
ke [label="Kenya"];
ug [label="Uganda"];
af [label="Africa"];
th [label="Thessaloniki"];
at [label="Athens"];
ch [label="Chios"];
bu [label="Bulgaria"];
se [label="Serbia"];
hu [label="Hungary"];
ge [label="Germany"];
ir [label="Iraq"];
it [label="Italia"];
ke -> af[color=red,penwidth=2.0];
ug -> af[color=red,penwidth=2.0];
sy -> tu -> sy[color=blue,penwidth=2.0];
g -> tu -> g[color=blue,penwidth=2.0];
at ->g[color=red,penwidth=2.0];
ch -> g[color=red,penwidth=2.0];
th ->g[color=red,penwidth=2.0];
m -> g -> m[color=blue,penwidth=2.0];
m -> se -> m[color=blue,penwidth=2.0];
m -> bu -> m[color=blue,penwidth=2.0];
hu -> se -> hu[color=blue,penwidth=2.0];
au -> hu -> au[color=blue,penwidth=2.0];
au -> ge -> au[color=blue,penwidth=2.0];
tu -> ir -> tu[color=blue,penwidth=2.0];
m -> se [dir="both",color=green,penwidth=2.0];
m -> bu [dir="both",color=green,penwidth=2.0];
hu -> au[color=green,penwidth=2.0];
au -> ge[color=green,penwidth=2.0];
tu-> it[color=green,penwidth=2.0];
tu -> g[color=green,penwidth=2.0];
ir -> tu[color=green,penwidth=2.0];
m -> th [dir="both",color=green,penwidth=2.0];
m -> ch [dir="both",color=green,penwidth=2.0];
m -> at [dir="both",color=green,penwidth=2.0];
g -> m[color=green,penwidth=2.0];
se -> hu[color=green,penwidth=2.0];
}
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment