jass package

Subpackages

• jass.test package
  • Submodules
  • jass.test.annotations_manager_test module
  • jass.test.storage_manager_test module
  • Module contents

Submodules

jass.annotations_manager module

class jass.annotations_manager.AnnotationManager

Bases: jass.storage_manager.StorageManager

ALL_STORAGE = 0

BASIC_BATCH_FORMAT = 0

{
    common: {

    }
    data : [{annotation1, ...}]
}

common : Elements contained in common will be copied for each annotation annotation1 : Annotation specific information

BATCH_STORAGE = 2

{data:[{annotation1}...{annotationN}]}
    Where: annotation1 is the same format as if you would only get one
           annotation

COMPACT_BATCH_FORMAT = 1

HUMAN_STORAGE = 1

SCORE_FIELD_NAME = '%textScore'

addStorageCollection(storageType, collectionName)

Since we may have different collections depending on the type of storage we need a keypair value.

createAnnotation(jsonDoc, strDocId, storageType=1)

This function creates an annotation. Currently this only works for annotations in storageType = 1

@Preconditions:

documentId : A valid storage id. (We don’t check that it exists

however). documentId will be added to the annotation object as doc_id (or overwrite existing field).

For the rest check createMongoDocument(self,jsonDoc,coll).

createAnnotationS(jsonBatch, strDocId, batchFormat=1, storageType=1)

Inserts annotations by batch. All annotations must be valid. Raises an error if there is even a single invalid annotation.

A valid annotation after processing has the following attributes:

doc_id:Describes the id of the document containing the annotation. Equals to strDocId. @context:A field linking the context of the document.

This field will be automatically created. _id: A unique id identifying the annotation,

:@param jsonBatch : JSON of the message. See batch format on how this

field is supposed to be structured.

:@param strDocId : Id of the document containing the annotation :@param storageType : Describes how to store the elements. (Currently can not be changed) Supports: 1,2

:@param batchFormat : Describes the format of the elements to input.

Supports: 0,1

@return: Number of created annotations.

deleteAnnotation(strAnnoId, storageType=1)

Currently this only works for annotations in storageType = 1

See deleteMongoDocument(self,strDocId,coll) for more info.

deleteAnnotationS(documentIds, jsonSelect={}, storageType=1)

Delete multiple annotations.

@param documentIds:  List of documents containing the annotations. @param jsonSelect:  Additional query parameters, which can restrict the search: See: http://docs.mongodb.org/manual/reference/operator/query/ for options @param storageType:  Describe which annotation storage to search. Supports: 0,1,2 @param batchFormat:  Describes how the elements would be returned Supports : 0 @return:Number of documents deleted.

getAnnotation(strAnnoId, storageType=1)

Currently this only works for annotations in storageType = 1

See getMongoDocument(self,strDocId,coll).

getAnnotationS(documentIds, jsonSelect={}, batchFormat=0, storageType=0)

Returns annotations respecting search criteria

@param documentIds:  List of documents containing the annotations. @param jsonSelect:  Additional query parameters, which can restrict the search: See: http://docs.mongodb.org/manual/reference/operator/query/ for options @param storageType:  Describe which annotation storage to search. Supports: 0,1,2 @param batchFormat:  Describes how the elements would be returned Supports : 0 @return:Documents found. Return format is described by batchFormat.

get_text_index_fields() → list

grouped_search_annotations(query: str, skip: int, limit: int) → dict

Search manual annotations(storageType 1) and group them by timeline. The body of the request is a JSON query passed to MongoDb collection aggregate method. https://docs.mongodb.com/manual/reference/method/db.collection.aggregate/ Skip & limit apply to the annotation list of each group.

Parameters:

• query – JSON query passed to MongoDb $match (Should be a $text search
• skip – The number of result to skip.
• limit – The maximum number of results to return

Returns:

Array of annotation matches grouped by timeline (annotationSetId). Each match contains the annotation and score matching the query, sorted descending by score. Groups are also sorted descending by score.

replaceAnnotationsInLargeStorage(jsonBatch, strDocId, batchFormat=1)

Create or replace batch annotations. Only works for batches present in large storage. This will first delete any batches (either the default batch, or batches returned by “common” section. Then it will create the new batch.

see createAnnotationS, deleteAnnotationS for more details.

search_annotations(query: str, skip: int, limit: int) → dict

Search manual annotations (storageType 1) The body of the request is a JSON query passed to MongoDb collection find method. https://docs.mongodb.com/manual/reference/method/db.collection.find/ Skip & limit can be used to implement pagination of the results.

Parameters:

• query – JSON query passed to MongoDb find
• skip – The number of result to skip.
• limit – The maximum number of results to return

Returns:

Array of results containing the annotation and score matching the query, sorted descending by score.

updateAnnotation(jsonDoc, strDocId, storageType=1)

This function updates an annotation. Currently this only works for annotations in storageType = 1.

@Preconditions:

documentId : A valid storage id. (we don’t check that it exists

however). strDocId will be added to the annotation object as doc_id (or overwrite existing field).

jass.custom_logger module

Custom logging module.

jass.custom_logger.logError(genException)

This function will take an object of the type GenericException and log it as an error.

jass.custom_logger.logInfo(genException)

This function will take an object of the type GenericException and log it as information useful for non critical errors.

jass.custom_logger.logUnknownDebug(context, msg)

A simple function to display a debug into logs

Parameters:

• msg – Custom error message to put
• context – The context to categorize the message

jass.custom_logger.logUnknownError(context, msg, e)

This function will log all non custom exceptions

Parameters:

• msg – Custom error message to put
• context – The context to categorize the message
• e – Exception object

jass.custom_logger.logUnknownInfo(context, msg)

A simple function to display a info into logs

Parameters:

• msg – Custom error message to put
• context – The context to categorize the message

jass.custom_logger.logUnknownWarning(context, msg)

A simple function to display a warning into logs

Parameters:

• msg – Custom error message to put
• context – The context to categorize the message

jass.document_manager module

class jass.document_manager.DocumentManager

Bases: jass.storage_manager.StorageManager

deleteDocumentWithContents(documentId)

Delete document and all related annotations

jass.error module

class jass.error.Error

Bases: object

ANNOTATION_SOURCE_NOT_VALID = -10

BAD_SERVICE_CONFIGURATION = -7

DB_READING_ERROR = -6

DB_WRITING_ERROR = -5

FILE_NOT_FOUND = -4

MISSING_PARAMETER = -11

NOT_INITIALIZED = -2

NO_ERROR = 0

SERVICE_NOT_FOUND = -8

UNKNOWN = -1

UNKNOWN_UUID = -9

URL_NOT_VALID = -3

get_html_status()

get_html_status_msg()

static get_html_status_msg_from_status(status)

get_message()

is_ok()

set_error(code)

set_error_with_details(code, **kwargs)

jass.generic_exception module

exception jass.generic_exception.GenericException(code, *args)

Bases: Exception

Exception wrapper which defines custom functionality for the JASS.

code = 0

context = 'GenericException'

jass.mongo_utils module

Various utilities for mongoDB usage.

jass.mongo_utils.changeDocIdToMongoId(jsonDoc)

Changes the _id to ObjectId. Will crash if jsonDoc is not a simple JSON object with _id field

jass.mongo_utils.changeDocIdToString(mongoDoc)

Changes the _id to string. Will crash if mongoDoc is not a valid Mongo Document

jass.mongo_utils.isObjectId(strId)

jass.reverse_proxied module

class jass.reverse_proxied.ReverseProxied(app)

Bases: object

Obtained from flask: http://flask.pocoo.org/snippets/35/

Wrap the application in this middleware and configure the front-end server to add these headers, to let you quietly bind this to a URL other than / and to an HTTP scheme that is different than what is used locally.

In nginx:

location /myprefix {
    proxy_pass http://192.168.0.1:5001;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Scheme $scheme;
    proxy_set_header X-Script-Name /myprefix;
    }

Parameters:app – the WSGI application

jass.settings module

General settings for the REST API

jass.settings.GetConfigValue(namespace, key)

Generic accessor for configuration values

Parameters:

• namespace – Section of the INI
• key – Actual key for which we want a value.

exception jass.settings.SettingsExceptions(code, *args)

Bases: jass.generic_exception.GenericException

Custom exceptions for settings

codeToMessage = {1: 'Cannot load settings because the configuration file is not found : {0}.', 3: 'Cannot load settings because the configuration file ({0}) loads an invalid services configuration file : {1}. Services configuration file must contains only one section being the service name.'}

jass.simple_rest module

This module defines a REST API for the annotation storage service as defined by the CANARIE API specification.

ANNOTATIONS

There are several common concepts shared used in annotations, which we should describe here

reservedFields:

fields: “_id” is reserved and will be ignored if passed either inside annotation or document.

storageType: Describe with which annotation storage we are

interacting. Multiple storages are required

Depending on the access of annotations.

Values:

0: All storage types

1: Human Storage : Storage used for small amount

of annotations made by humans.

Fast search, insert, delete,

update.(Great performance degradation,

if more then 10 millions annotations).

2: Large Storage : Storage used for large amount

of annotations (like automatic

annotations), which are not meant

to be used individually. Very fast search,

but can only operate on groups of annotations

on fields defined in common fields).

If no common field supplied.

batchFormat: Describe the format in which multiple annotations

should be send to the system. This also includes

how multiple annotations should be extracted from

the system.

Values:

0: {“data” : [{annotation1}…{annotationN}]}

Where: annotation1 is the same format as if

you would request 1 annotation

1: {

common: {

}

data : [{annotation1, …}]

}

common : elements contained in common will be

copied for each annotation

annotation1: annotation specific information

jsonSelect: A jsonQuery to get a subset of annotations. See

for query variables:

See http://docs.mongodb.org/manual/reference/operator/query/

for options

ERROR CODES

Here is the prefix by type:

Exception Type	Starting Code	Reference Document
UnknownException	50000	Unknown server crash error
StorageRestExceptions	50100	storage_exception.StorageRestExceptions
AnnotationException	51000	storage_exception.AnnotationException
MongoDocumentException	52000	storage_exception.MongoDocumentException
StorageException	53000	storage_exception.StorageException

jass.simple_rest.annotationSchema(schema_id)

Route:/annotationSchema/<schema_id>

Used to store annotation schemas. Same implementation as for the document.

jass.simple_rest.createAnnotationSchema()

Route:/annotationSchema

Used to store Annotation schemas Same implementation as for the document.

jass.simple_rest.createDocument()

Route:

/document

POST creates a new document:  

Request:

Preconditions: Here are the minimum required elements by the document.
{
    @context: context describing the format of the document
}

All the other parameters will be saved as is. Erases “_id”, “id” fields if they exists.

Response:

Same as in, plus creates a field “id” to identify the document

http status code:  

OK: 200

Error: See Error Codes

jass.simple_rest.createDocumentAnnotation(document_id)

Route:

/document/<document_id>/annotation

Parameters:

document_id – The id of the document for which we want to access the annotation

POST Creates an annotation.:  

Request:

preconditions:

Here are minimum annotations contents:

{
  @context: Complex object containing JSON_LD info.
}

Other custom fields which were created would be returned too.

The annotation using this method is created in HumanStorage.

Response JSON:

Here are minimum annotations contents which will be after creation:

{
   doc_id: to describe the id of the document containing the
           annotation. Equals to strDocId.
   @context: a field linking the context of the document.
   id:  a unique id identifying the annotation.
}

http status code:  

OK: 200

Error: See Error Codes

jass.simple_rest.document(document_id)

Route:/document/<document_id>

Get/Put/Delete for the documents.

Parameters:

document_id – The id of the document we want to access

GET returns a document:  

Response JSON:

Here are minimum document contents:

{
  id:                    Id of the document = document_id
  @context:              Complex object containing JSON_LD
                         info.
}

Other custom fields which were created would be returned too.

http status code:  

OK: 200

Error: See Error Codes

PUT updates a document by replacing whole document contents:  

Request:

The document must exists and contains the following contents at minimum:

{
    id:                    Id of the document = document_id
    @context:              Complex object containing JSON_LD
                           info.
}

Other custom fields which were created would be saved too. Erases “_id” field.

Response:

http status code:  

OK: 200

Error: See Error Codes

DELETE deletes the document. If the document not found do nothing.:  

Response JSON:

{}

http status code:  

OK: 200

Error: See Error Codes

jass.simple_rest.documentAnnotation(document_id, annotation_id)

Route:/document/<document_id>/annotation/<annotation_id>

Get/Update/Delete an annotation:

Parameters:

• document_id – The id of the document for which we want to access the annotation
• annotation_id – The id of the annotation we want to access

GET Returns an annotation:  

Request:

precondtions:Can only get annotations from HumanStorage.

Response json:

Here are minimum annotations contents which will be after creation:

{
    doc_id: to describe the id of the document containing the
            annotation. Equals to strDocId.
    @context: a field linking the context of the document.
    id:  a unique id identifying the annotation.
}

PUT Updates an annotation:  

Updates are made by changing the content of the old annotation with the new one

Request:

precondtions:Can only get annotations from HumanStorage.

Response:

http status code:  

OK: 200

Error: See Error Codes

DELETE deletes an annotation.:  

Request:

precondtions:Can only get annotations from HumanStorage.

Response:

http status code:  

OK: 204

Error: See Error Codes

jass.simple_rest.documentAnnotationS(document_id)

Route:/document/<document_id>/annotations

In case storageType = 2, annotations will be stored in batches. All operations impact each batch (Example a PUT operation will replace all annotations in a batch). By default each document have 1 annotation batch. To have multiple batches, post/put batch of annotations using batchFormat = 1, and use common section to to identify uniquely a batch. Fields “doc_id_batch”, “file_fs_id_batch” are reserved, thus will be ignored if added in common section.

Parameters:

document_id – The id of the document from which we want to access multiple annotations

POST Create annotations in batch.:  

Request:

preconditions:All must be valid annotations. If one annotation fails, it will return an error message and fail all create. params supported:  

batchFormat = 0,1

storageType = 1,2

params default:

batchFormat = 1

storageType = 1

Response json:

returns {“nInserted”:nbAnnotationsInserted}

http status code:  

OK: 200

Error: See Error Codes

PUT Updates annotations related for the document.:  

Request:

When we update we replace old contents with new ones. jsonSelect

params supported:  

jsonSelect (only contains contents in the “common” fields for storageType = 2.)

storageType = 2

params default:

storageType = 2

jsonSelect = {}

Response json:

Returns number of annotations deleted

{“nDeleted”:nbAnnotationsDeleted}

http status code:  

OK: 200

Error: See Error Codes

DELETE Deletes annotations related for the document.:  

Request:

params supported:  

jsonSelect

storageType = 1,2

params default:

storageType = 1

jsonSelect = {}

Response json:

Returns number of annotations deleted

{“nDeleted”:nbAnnotationsDeleted}

http status code:  

OK: 200

Error: See Error Codes

GET Returns annotations for the current document.:  

Request:

params supported:  

jsonSelect

storageType = 0,1,2

batchFormat = 0

params default:

batchFormat = 0

storageType = 0

jsonSelect = {}

Response json:

An array of annotations check batch format for how they will be formatted.

http status code:  

OK: 200

Error: See Error Codes

jass.simple_rest.home()

Return the home page for a particular service

jass.simple_rest.info()

Required by CANARIE

jass.simple_rest.internal_error400(error)

jass.simple_rest.internal_error500(error)

jass.simple_rest.log_request()

jass.simple_rest.search_annotations()

Search manual annotations (storageType 1) The body of the request is a JSON query passed to https://docs.mongodb.com/manual/reference/method/db.collection.find/

Returns:JSON Array of results containing the annotation and score matching the query, sorted descending by score.

jass.simple_rest.search_annotations_grouped()

Search manual annotations (storageType 1) and group them by timeline. The body of the request is a JSON query passed to https://docs.mongodb.com/manual/reference/method/db.collection.aggregate/

Limit is mandatory when skip is specified.

Returns:The text index fields and an array of annotation matches grouped by timeline (annotationSetId). Each match contains the annotation and score matching the query, sorted descending by score. Groups are also sorted descending by score.

jass.simple_rest.simple_requests_handler(api_request)

Handle simple requests required by CANARIE

jass.simple_rest.stats()

Required by CANARIE.

jass.singleton module

class jass.singleton.Singleton(decorated)

Bases: object

A non-thread-safe helper class to ease implementing singletons. This should be used as a decorator – not a metaclass – to the class that should be a singleton.

The decorated class can define one __init__ function that takes only the self argument. Other than that, there are no restrictions that apply to the decorated class.

To get the singleton instance, use the Instance method. Trying to use __call__ will result in a TypeError being raised.

Limitations: The decorated class cannot be inherited from.

Source: http://stackoverflow.com/questions/42558/python-and-the-singleton-pattern

Instance()

Returns the singleton instance. Upon its first call, it creates a new instance of the decorated class and calls its __init__ method. On all subsequent calls, the already created instance is returned.

jass.storage_exception module

exception jass.storage_exception.AnnotationException(code, *args)

Bases: jass.generic_exception.GenericException

Exceptions when manipulating Annotations

codeToMessage = {0: 'Unexpected Exception', 1: 'Invalid document object Id {0}', 2: 'Annotation: {0} in batch is missing a context for doc {1}', 3: 'Invalid document id detected: {0}, operation aborted.', 4: 'Annotation: {0} in batch is missing a required field for doc {1}', 5: 'Format {0} is not supported', 6: 'StorageType: {0} is not initialized ', 7: 'StorageType: {0} is not supported ', 8: 'Number of inserted annotations is not equal to the number of annotations in batch: {0} vs {1}', 9: 'Some of the document IDs have invalid format.'}

context = 'Annotation Storage Annotation'

exception jass.storage_exception.MongoDocumentException(code, *args)

Bases: jass.generic_exception.GenericException

Exceptions occurring when Manipulating an object in MongoDB,

codeToMessage = {0: 'Unexpected Exception', 2: 'Storage document should not be empty.', 3: 'Storage document is missing the required @context field', 4: 'Storage document contains a reserved field : {0}', 5: 'Storage Document with id : {0} not found for update'}

context = 'Annotation Storage Document'

exception jass.storage_exception.StorageException(code, *args)

Bases: jass.generic_exception.GenericException

Exceptions related to the functionality of the MongoDB database itself

codeToMessage = {1: 'Failed to connect to MongoDB', 2: 'Unexpected delete fail to MongoDB'}

context = 'Annotation Storage Storage'

exception jass.storage_exception.StorageRestExceptions(code, *args)

Bases: jass.generic_exception.GenericException

Exception related to the rest part of the program

codeToMessage = {0: 'ErrorCode not supported. Please contact administrator if you see this error.', 1: 'Inconsistent document. The id contained in the document is not the same as the one called.', 2: 'Did not found the document requested', 3: 'Can not update document since the id described does not exist.', 4: 'The id supplied is not located in Human Storage. This request only works with Human Storage.', 5: 'One of the supplied request parameters is invalid.'}

jass.storage_manager module

class jass.storage_manager.StorageManager

Bases: object

Storage manager class.

aggregate(pipeline: list, collection: str, **kwargs)

Aggregation pipeline.

Implementation detail: this will pass the allowDiskTrue to MongoDB.

@param documentIds:  List of document IDs for which we should search annotations. @param A queryToDetermine how to select annotations:  see http://docs.mongodb.org/manual/reference/operator/query/ for options

:@return the number of deleted documents

connect()

Connects to MongoDB and verifies connection.

createMongoDocument(jsonDoc, collection=None)

Creates a new document. This function only validates the presence of the required fields.

Preconditions (Otherwise exception is thrown):  

• isConnected must be true,
• jsonDoc must exist and be a valid JSON object,

Parameters:

jsonDoc –

Contents of the document as string

Here are the elements required by the document:

{
    @context: context describing the format of the document
}

If the document contains the field _id, the _id field will be deleted and another _id field will be generated instead. The generated _id will be required to access the document.

param collection:  Enables you to override the default collection if needed return _id:The ID of the created document

deleteMongoDocument(strDocId, collection=None)

Deletes a document specified by ID.

Parameters:strDocId – Document ID as string. Should be unique. Returns:0 if no elements were deleted, 1 if one was deleted.

deleteMongoDocumentS(jsonQuery, collection=None)

Delete multiple annotations.

@param documentIds:  List of document IDs which should be affected. @param a queryToDetermine how to select annotations:  see http://docs.mongodb.org/manual/reference/operator/query/ for options

:@return The number of deleted documents

disconnect()

getMongoDocument(strDocId, collection=None)

Returns a document This function only validates the presence of the required fields.

Preconditions (Otherwise exception is thrown):  

• isConnected must be true,

Parameters:

strDocId – Document ID

:return : If the document is found returns a json object of the

document, otherwise returns None

Document content returned (mandatory). Other user fields may be present:

{
    _id: Document id as a string
    @context: context describing the format of the document
}

getMongoDocumentS(jsonQuery, collection=None, **kwargs)

Search a collection for documents.

@param documentIds:  List of document IDs for which we should search annotations. @param A queryToDetermine how to select annotations:  see http://docs.mongodb.org/manual/reference/operator/query/ for options

:@return the number of deleted documents

static grouped_annotation_filter(limit, skip)

isConnected()

Connects to MongoDB and verifies connection.

m_connected = False

setCollection(collection)

Collection in which documents will be created/updated/destroyed.

updateMongoDocument(jsonDoc, collection=None)

Updates an existing document, by replacing it with new contents. This function only validates the presence of the required fields.

Preconditions (Otherwise exception is thrown):  

• isConnected must be true,
• required fields must be present

Parameters:

jsonDoc – Document as a JSON document. The document needs to contain a valid id.

:return : If the document to be updated is found, returns the id of the document. If it can not be found, raises an exception.

Document content returned (mandatory). Other user fields may be present:

{
    _id: Document id as a string
    @context: Context describing the format of the document
}

jass.storage_manager.clear_nones(a_dict: dict) → dict

jass.utility_rest module

This module is a collection of utility functions used by the rest_route module placed here to keep the rest_route module as clear as possible.

exception jass.utility_rest.UnknownServiceError

Bases: Exception

Service name is of unknown type

jass.utility_rest.error_response(status, message, code, more_info)

Format error according to request type.

jass.utility_rest.get_canarie_api_response(service_name, template_path, canarie_api_request)

Provide a valid HTML response for the CANARIE API request based on the service_route.

Param:

service_route:Route name of the service coming from the URL e.g.: [‘diarisation’, ‘STT’, etc.] canarie_api_request:  The request specified in the URL

Returns:

A valid html response

jass.utility_rest.get_server_restart_time()

Obtain the server status provided by Apache if properly configured. The configuration file must contain the following section:

# Enable the module mod_status
# See : http://httpd.apache.org/docs/2.2/mod/mod_status.html
<Location /server-status>
    SetHandler server-status
    Order deny,allow
    #Deny access for everyone outside
    Deny from all
    #Allow access from IP 10* (private address inside local network)
    Allow from 10
    #Allow access from 132.217* (public IP range own by the CRIM)
    Allow from 132.217
</Location>

# Keep track of extended status information for each request
ExtendedStatus On

jass.utility_rest.handle_error(more_info, status)

Creates a standardized error response.

jass.utility_rest.make_error_response(error_code)

Create an error response from a single error code.

jass.utility_rest.make_error_response_with_details(error_code, **kwargs)

Create an error response from a single error code.

jass.utility_rest.request_wants_json()

Check if the request type is JSON

Deals with /

Module contents