lofn.base package

Submodules

lofn.base.hdfs module

Base classes and functions to interact with HDFS.

lofn.base.hdfs.get(remote, local)

Get from HDFS

lofn.base.hdfs.mkdir_p(path)

Make directory with parents if they don’t exist

lofn.base.hdfs.put(local, remote)

Put into HDFS

lofn.base.hdfs.rm_r(path)

Remove recursively

lofn.base.tmp module

Base functions and classes for using temp files.

lofn.base.tmp.create_temp_directory(directory=None)

Create a temporary directory. Supports setting a parent directory that is not temporary and will try to create it if it does not already exist.

Parameters:directory – specify a parent directory instead of using default (None -> /tmp) Returns:path to directory

lofn.base.tmp.write_binary_to_temp_file(data, path)

Write string directly to file for binary data.

lofn.base.tmp.write_to_temp_file(iterable, path)

Write iterable to temporary file, an item per line.

Parameters:

• iterable – list or tuples of contents to write to file
• path – absolute path to file. We use a tempdir as parent.

Returns:

path to temporary file