untitaker/ python-atomicwrites

Python

posix

atomic

Need help with python-atomicwrites?

Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

untitaker

267 Stars

32 Forks

MIT License

131 Commits

4 Opened issues

Description

Powerful Python library for atomic file writes.

Services available

Need anything else?

Contributors list

Readme

===================

python-atomicwrites

Name: python-atomicwrites
Brand: python-atomicwrites
SKU: //untitaker/python-atomicwrites
Rating: 2.4175 (267 reviews)

.. image:: https://travis-ci.com/untitaker/python-atomicwrites.svg?branch=master :target: https://travis-ci.com/untitaker/python-atomicwrites .. image:: https://ci.appveyor.com/api/projects/status/vadc4le3c27to59x/branch/master?svg=true :target: https://ci.appveyor.com/project/untitaker/python-atomicwrites/branch/master .. image:: https://readthedocs.org/projects/python-atomicwrites/badge/?version=latest :target: https://python-atomicwrites.readthedocs.io/en/latest/?badge=latest :alt: Documentation Status

Atomic file writes.

.. code-block:: python

from atomicwrites import atomic_write

with atomic_write('foo.txt', overwrite=True) as f:
    f.write('Hello world.')
    # "foo.txt" doesn't exist yet.
Now it does.

See

API documentation

_ for more low-level interfaces.

Features that distinguish it from other similar libraries (see

Alternatives and Credit

_):

Race-free assertion that the target file doesn't yet exist. This can be controlled with the
```
overwrite
```
parameter.
Windows support, although not well-tested. The MSDN resources are not very explicit about which operations are atomic. I'm basing my assumptions off
```
a
comment
```
_ by
```
Doug Crook
```
_, who appears to be a Microsoft employee:
Question: Is MoveFileEx atomic if the existing and new files are both on the same drive?

The simple answer is "usually, but in some cases it will silently fall-back to a non-atomic method, so don't count on it".

The implementation of MoveFileEx looks something like this: [...]

The problem is if the rename fails, you might end up with a CopyFile, which is definitely not atomic.

If you really need atomic-or-nothing, you can try calling NtSetInformationFile, which is unsupported but is much more likely to be atomic.
Simple high-level API that wraps a very flexible class-based API.
Consistent error handling across platforms.

How it works

It uses a temporary file in the same directory as the given path. This ensures that the temporary file resides on the same filesystem.

The temporary file will then be atomically moved to the target location: On POSIX, it will use

rename

if files should be overwritten, otherwise a combination of

link

and

unlink

. On Windows, it uses MoveFileEx_ through stdlib's

ctypes

with the appropriate flags.

Note that with

link

and

unlink

, there's a timewindow where the file might be available under two entries in the filesystem: The name of the temporary file, and the name of the target file.

Also note that the permissions of the target file may change this way. In some situations a

chmod

can be issued without any concurrency problems, but since that is not always the case, this library doesn't do it by itself.

.. _MoveFileEx: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365240%28v=vs.85%29.aspx

fsync

On POSIX,

fsync

is invoked on the temporary file after it is written (to flush file content and metadata), and on the parent directory after the file is moved (to flush filename).

fsync

does not take care of disks' internal buffers, but there don't seem to be any standard POSIX APIs for that. On OS X,

fcntl

is used with

F_FULLFSYNC

instead of

fsync

for that reason.

On Windows,

_commit

_ is used, but there are no guarantees about disk internal buffers.

Alternatives and Credit

Atomicwrites is directly inspired by the following libraries (and shares a minimal amount of code):

The Trac project's
```
utility functions
```
, also used in
Werkzeug
and
```
mitsuhiko/python-atomicfile
```
_. The idea to use
```
ctypes
```
instead of
```
PyWin32
```
originated there.
```
abarnert/fatomic 
```
_. Windows support (based on
```
PyWin32
```
) was originally taken from there.

Other alternatives to atomicwrites include:

```
sashka/atomicfile 
```
_. Originally I considered using that, but at the time it was lacking a lot of features I needed (Windows support, overwrite-parameter, overriding behavior through subclassing).
The
```
Boltons library collection 
```
_ features a class for atomic file writes, which seems to have a very similar
```
overwrite
```
parameter. It is lacking Windows support though.

License

Licensed under the MIT, see

LICENSE