org.apache.tephra

Class TransactionManager

java.lang.Object

com.google.common.util.concurrent.AbstractService

org.apache.tephra.TransactionManager

All Implemented Interfaces:

com.google.common.util.concurrent.Service

public class TransactionManager
extends com.google.common.util.concurrent.AbstractService

This is the central place to manage all active transactions in the system. A transaction consists of

• A write pointer: This is the version used for all writes of that transaction.
• A read pointer: All reads under the transaction use this as an upper bound for the version.
• A set of excluded versions: These are the write versions of other transactions that must be excluded from reads, because those transactions are still in progress, or they failed but couldn't be properly rolled back.

To use the transaction system, a client must follow this sequence of steps:

1. Request a new transaction.
2. Use the transaction to read and write datasets. Datasets are encouraged to cache the writes of the transaction in memory, to reduce the cost of rollback in case the transaction fails.
3. Check whether the transaction has conflicts. For this, the set of change keys are submitted via canCommit(), and the transaction manager verifies that none of these keys are in conflict with other transactions that committed since the start of this transaction.
4. If the transaction has conflicts:
   1. Roll back the changes in every dataset that was changed. This can happen in-memory if the changes were cached.
   2. Abort the transaction to remove it from the active transactions.
5. If the transaction has no conflicts:
1. Persist all datasets changes to storage.
2. Commit the transaction. This will repeat the conflict detection, because more overlapping transactions may have committed since the first conflict check.
3. If the transaction has conflicts:
1. Roll back the changes in every dataset that was changed. This is more expensive because changes must be undone in persistent storage.
2. Abort the transaction to remove it from the active transactions.

Transactions may be short or long-running. A short transaction is started with a timeout, and if it is not committed before that timeout, it is invalidated and excluded from future reads. A long-running transaction has no timeout and will remain active until it is committed or aborted. Long transactions are typically used in map/reduce jobs and can produce enormous amounts of changes. Therefore, long transactions do not participate in conflict detection (they would almost always have conflicts). We also assume that the changes of long transactions are not tracked, and therefore cannot be rolled back. Hence, when a long transaction is aborted, it remains in the list of excluded transactions to make its writes invisible.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	TransactionManager.ChangeSet Represents a set of changes from a client.
static class	TransactionManager.InProgressTx Represents some of the info on in-progress tx
static class	TransactionManager.InProgressType Type of in-progress transaction.

Nested classes/interfaces inherited from interface com.google.common.util.concurrent.Service

com.google.common.util.concurrent.Service.Listener, com.google.common.util.concurrent.Service.State

Constructor Summary

Constructors

Constructor and Description
TransactionManager(org.apache.hadoop.conf.Configuration config)
TransactionManager(org.apache.hadoop.conf.Configuration conf, TransactionStateStorage persistor, MetricsCollector txMetricsCollector)

Method Summary

Modifier and Type	Method and Description
void	abort(Transaction tx)
void	canCommit(long txId, Collection<byte[]> changeIds)
Transaction	checkpoint(Transaction originalTx)
void	commit(long txId, long writePointer)
void	doStart()
void	doStop()
TransactionSnapshot	getCurrentState()
int	getExcludedListSize()
int	getInvalidSize()
TransactionSnapshot	getSnapshot()
TransactionStateStorage	getTransactionStateStorage()
static long	getTxExpirationFromWritePointer(long writePointer, long timeoutInSeconds)
boolean	invalidate(long tx)
void	logStatistics() Called from the tx service every 10 seconds.
void	recoverState()
void	resetState() Resets the state of the transaction manager.
Transaction	startLong() Start a long transaction.
Transaction	startLong(String clientId) Starts a long transaction with a client id.
Transaction	startShort() Start a short transaction with the default timeout.
Transaction	startShort(int timeoutInSeconds) Start a short transaction with a given timeout.
Transaction	startShort(String clientId) Start a short transaction with a client id and default timeout.
Transaction	startShort(String clientId, int timeoutInSeconds) Start a short transaction with a given timeout.
boolean	takeSnapshot(OutputStream out) Take a snapshot of the transaction state and serialize it into the given output stream.
boolean	truncateInvalidTx(Set<Long> invalidTxIds) Removes the given transaction ids from the invalid list.
boolean	truncateInvalidTxBefore(long time) Removes all transaction ids started before the given time from invalid list.
static Map<Long,TransactionManager.InProgressTx>	txnBackwardsCompatCheck(int defaultLongTimeout, long longTimeoutTolerance, Map<Long,TransactionManager.InProgressTx> inProgress) Check if in-progress transactions need to be migrated to have expiration time and type, if so do the migration.

Methods inherited from class com.google.common.util.concurrent.AbstractService

addListener, isRunning, notifyFailed, notifyStarted, notifyStopped, start, startAndWait, state, stop, stopAndWait, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

TransactionManager

public TransactionManager(org.apache.hadoop.conf.Configuration config)

TransactionManager

@Inject
public TransactionManager(org.apache.hadoop.conf.Configuration conf,
                                  @Nonnull
                                  TransactionStateStorage persistor,
                                  MetricsCollector txMetricsCollector)

Method Detail

doStart

public void doStart()

Specified by:

doStart in class com.google.common.util.concurrent.AbstractService

getSnapshot

public TransactionSnapshot getSnapshot()
                                throws IOException

Throws:

IOException

takeSnapshot

public boolean takeSnapshot(OutputStream out)
                     throws IOException

Take a snapshot of the transaction state and serialize it into the given output stream.

Returns:

whether a snapshot was taken.

Throws:

IOException

getCurrentState

public TransactionSnapshot getCurrentState()

recoverState

public void recoverState()

txnBackwardsCompatCheck

public static Map<Long,TransactionManager.InProgressTx> txnBackwardsCompatCheck(int defaultLongTimeout,
                                                                                long longTimeoutTolerance,
                                                                                Map<Long,TransactionManager.InProgressTx> inProgress)

Check if in-progress transactions need to be migrated to have expiration time and type, if so do the migration. This is required for backwards compatibility, when long running transactions were represented with expiration time -1. This can be removed when we stop supporting SnapshotCodec version 1.

resetState

public void resetState()

Resets the state of the transaction manager.

doStop

public void doStop()

Specified by:

doStop in class com.google.common.util.concurrent.AbstractService

startShort

public Transaction startShort()

Start a short transaction with the default timeout.

startShort

public Transaction startShort(String clientId)

Start a short transaction with a client id and default timeout.

Parameters:

clientId - id of the client requesting a transaction.

startShort

public Transaction startShort(int timeoutInSeconds)

Start a short transaction with a given timeout.

Parameters:

timeoutInSeconds - the time out period in seconds.

startShort

public Transaction startShort(String clientId,
                              int timeoutInSeconds)

Start a short transaction with a given timeout.

Parameters:

clientId - id of the client requesting a transaction.

timeoutInSeconds - the time out period in seconds.

getTxExpirationFromWritePointer

public static long getTxExpirationFromWritePointer(long writePointer,
                                                   long timeoutInSeconds)

startLong

public Transaction startLong()

Start a long transaction. Long transactions and do not participate in conflict detection. Also, aborting a long transaction moves it to the invalid list because we assume that its writes cannot be rolled back.

startLong

public Transaction startLong(String clientId)

Starts a long transaction with a client id.

canCommit

public void canCommit(long txId,
                      Collection<byte[]> changeIds)
               throws TransactionNotInProgressException,
                      TransactionSizeException,
                      TransactionConflictException

Throws:

TransactionNotInProgressException

TransactionSizeException

TransactionConflictException

commit

public void commit(long txId,
                   long writePointer)
            throws TransactionNotInProgressException,
                   TransactionConflictException

Throws:

TransactionNotInProgressException

TransactionConflictException

abort

public void abort(Transaction tx)

invalidate

public boolean invalidate(long tx)

truncateInvalidTx

public boolean truncateInvalidTx(Set<Long> invalidTxIds)

Removes the given transaction ids from the invalid list.

Parameters:

invalidTxIds - transaction ids

Returns:

true if invalid list got changed, false otherwise

truncateInvalidTxBefore

public boolean truncateInvalidTxBefore(long time)
                                throws InvalidTruncateTimeException

Removes all transaction ids started before the given time from invalid list.

Parameters:

time - time in milliseconds

Returns:

true if invalid list got changed, false otherwise

Throws:

InvalidTruncateTimeException - if there are any in-progress transactions started before given time

checkpoint

public Transaction checkpoint(Transaction originalTx)
                       throws TransactionNotInProgressException

Throws:

TransactionNotInProgressException

getExcludedListSize

public int getExcludedListSize()

getInvalidSize

public int getInvalidSize()

Returns:

the size of invalid list

logStatistics

public void logStatistics()

Called from the tx service every 10 seconds. This hack is needed because current metrics system is not flexible when it comes to adding new metrics.

getTransactionStateStorage

public TransactionStateStorage getTransactionStateStorage()