ds.granules.operation

Class MapReduceBase

java.lang.Object

ds.granules.operation.OperationBase

ds.granules.operation.MapReduceBase

All Implemented Interfaces:

ds.funnel.data.format.FormatMarshaller, Map, MapReduce, Operation, OperationConfiguration, Reduce

Direct Known Subclasses:

StreamBase

public abstract class MapReduceBase
extends OperationBase
implements Map, Reduce

This is the base class which will serve as the base for implementations of the Map and Reduce interface.

Author:

Shrideep Pallickara

Nested Class Summary

Nested classes/interfaces inherited from interface ds.granules.operation.Operation

Operation.OperationType

Field Summary

Fields

Modifier and Type	Field and Description
protected ds.funnel.topic.Topic	defaultOutputStream
protected StreamDataset	mapReduceDataset
static String	MODULE

Constructor Summary

Constructors

Constructor and Description
MapReduceBase()

Method Summary

Methods

Modifier and Type	Method and Description
int	addMap(Map map) Adds a Map that will be reduced.
protected void	addMapper(Map map) This method is called on a Reduce, once it has been added to a Map using the Map.addReduce() method.
int	addReduce(Reduce reduce) Adds a Reduce for this particular Map.
protected void	addReducer(Reduce reduce) This method is called on a Map, once it has been added to a Reduce using the Reduce.addMap() method.
Results	createResults(boolean completeResult, boolean processingComplete) Create the Results type to be used for reporting results produced as a result of the processing.
Iterator<String>	getAwaitingOutputsFrom() Retrieves the set of Map identifiers from which an Output has not been received so far.
DatasetCollection	getDatasetCollection() Retrieve the dataset that is associated with this particular operation.
Enumeration<String>	getMapIdentifiers() Retrieves the set of Map identifiers
protected StreamDataset	getMapReduceDataset()
int	getNumberOfMaps() Retrieves the number of registered Map instances
int	getNumberOfReducers() Retrieves the number of registered Reduce instances
Enumeration<String>	getReduceIdentifiers() Retrieves the set of Reduce identifiers
boolean	isAwaitingOutputs() Check to see is the Reducer is missing outputs from any of the registered Maps.
void	marshall(ds.funnel.data.format.FormatWriter mFormat)
int	removeMap(String mapIdentifier) Removes a Map from the list of Maps that are registered to this reducer.
protected void	removeMapper(Map map) This method is called on a Reduce, once it has been removed from a Map using the Map.removeReduce() method.
int	removeReduce(String reduceIdentifier) Removes a given reducer from the list of Reducers for a given Map function.
protected void	removeReducer(Reduce reduce) This method is called on a Map, once it has been removed from a Reduce using the Reduce.removeMap() method.
void	resetOutputTracker() This method resets the output tracker associated with the reducer.
void	setReceivedOutputFrom(String mapIdentifier) Mark the receipt of Output from a given Map instance
void	unmarshall(ds.funnel.data.format.FormatReader mFormat) Unmarshall the operation from a set of marshalled bytes.
void	writeResults(int synopsisType, String streamSynopsis, Results results) Deprecated.
void	writeResults(Results results) Writes the specified results to the default output stream.
void	writeResults(String streamSynopsis, Results results) Deprecated.
void	writeResults(ds.funnel.topic.Topic topic, Results results) Writes the specified results to a specific output stream.

Methods inherited from class ds.granules.operation.OperationBase

changeToExecuteFixedNumberOfTimes, changeToExecuteOnce, changeToExecutePeriodically, changeToExecuteWhenDataAvailable, changeToStayAlive, dispose, execute, getDomain, getExecutionProfile, getInstanceIdentifier, getOperationIdentifier, getProcessingDirectives, getVersionInformation, hasDatasetCollection, hasExecutionProfile, hasProcessingDirectives, initialize, isInitialized, markInitializationAsComplete, needsInitialization, setAsExecuteContinuously, setAsExecuteFixedNumberOfTimes, setAsExecuteOnce, setAsExecutePeriodically, setAsExecuteWhenDataAvailable, setAsStayAlive, setDatasetCollection, setDomain, setInitializationComplete, setInitializationNeeded, setInstanceIdentifier, setOperationIdentifier, setProcessingDirectives, setTeminationConditionReached, setVersionInformation, terminationConditionReached, toString

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Methods inherited from interface ds.granules.operation.Operation

dispose, execute, getDomain, getExecutionProfile, getInstanceIdentifier, getOperationIdentifier, getProcessingDirectives, getVersionInformation, hasDatasetCollection, hasExecutionProfile, hasProcessingDirectives, initialize, isInitialized, needsInitialization, terminationConditionReached, toString

Field Detail

MODULE

public static final String MODULE

See Also:

Constant Field Values

mapReduceDataset

protected StreamDataset mapReduceDataset

defaultOutputStream

protected ds.funnel.topic.Topic defaultOutputStream

Constructor Detail

MapReduceBase

public MapReduceBase()

Method Detail

unmarshall

public void unmarshall(ds.funnel.data.format.FormatReader mFormat)
                throws ds.funnel.data.format.FormatException

Description copied from interface: Operation

Unmarshall the operation from a set of marshalled bytes.

Specified by:

unmarshall in interface Operation

Overrides:

unmarshall in class OperationBase

Parameters:

mFormat - The byte[] representation of the operation.

Throws:

ds.funnel.data.format.FormatException

marshall

public void marshall(ds.funnel.data.format.FormatWriter mFormat)
              throws ds.funnel.data.format.FormatException

Specified by:

marshall in interface ds.funnel.data.format.FormatMarshaller

Overrides:

marshall in class OperationBase

Throws:

ds.funnel.data.format.FormatException

getDatasetCollection

public DatasetCollection getDatasetCollection()

Description copied from interface: Operation

Retrieve the dataset that is associated with this particular operation.

Specified by:

getDatasetCollection in interface Operation

Overrides:

getDatasetCollection in class OperationBase

Returns:

The underlying dataset collection.

getMapReduceDataset

protected StreamDataset getMapReduceDataset()

addReduce

public int addReduce(Reduce reduce)
              throws MapReduceException

Description copied from interface: Map

Adds a Reduce for this particular Map. Typically, a given Map instance will have only one Reduce. There are special instances when multiple Reducers would be needed for a given Map function.

Specified by:

addReduce in interface Map

Parameters:

reduce - The Reducer that will be added to this Map

Returns:

The number of reducers for this Map.

Throws:

MapReduceException - If there are problems in adding this Reduce

removeReduce

public int removeReduce(String reduceIdentifier)
                 throws MapReduceException

Description copied from interface: Map

Removes a given reducer from the list of Reducers for a given Map function.

Specified by:

removeReduce in interface Map

Parameters:

reduceIdentifier - The identifier of the Reduce to be removed

Returns:

The number of reducers for this Map, after the successful removal

Throws:

MapReduceException - If there are problems during removal.

getNumberOfReducers

public int getNumberOfReducers()

Description copied from interface: Map

Retrieves the number of registered Reduce instances

Specified by:

getNumberOfReducers in interface Map

Returns:

The number of reducers for this map.

getReduceIdentifiers

public Enumeration<String> getReduceIdentifiers()

Description copied from interface: Map

Retrieves the set of Reduce identifiers

Specified by:

getReduceIdentifiers in interface Map

Returns:

An enumeration of the list of registered identifiers

addMap

public int addMap(Map map)
           throws MapReduceException

Description copied from interface: Reduce

Adds a Map that will be reduced.

Specified by:

addMap in interface Reduce

Parameters:

map - Add a Map instance for this reducer

Returns:

The number of maps that are registered with this reducer

Throws:

MapReduceException

removeMap

public int removeMap(String mapIdentifier)
              throws MapReduceException

Description copied from interface: Reduce

Removes a Map from the list of Maps that are registered to this reducer.

Specified by:

removeMap in interface Reduce

Parameters:

mapIdentifier - The identifier of the Map to removed

Returns:

The number of maps that are registered with this reducer

Throws:

MapReduceException - If there are problems removing the Map.

getMapIdentifiers

public Enumeration<String> getMapIdentifiers()

Description copied from interface: Reduce

Retrieves the set of Map identifiers

Specified by:

getMapIdentifiers in interface Reduce

Returns:

An enumeration of the list of registered identifiers

getNumberOfMaps

public int getNumberOfMaps()

Description copied from interface: Reduce

Retrieves the number of registered Map instances

Specified by:

getNumberOfMaps in interface Reduce

Returns:

The number of Maps

isAwaitingOutputs

public boolean isAwaitingOutputs()

Description copied from interface: Reduce

Check to see is the Reducer is missing outputs from any of the registered Maps.

Specified by:

isAwaitingOutputs in interface Reduce

Returns:

true is the Reducer is still awaiting outputs; false otherwise.

setReceivedOutputFrom

public void setReceivedOutputFrom(String mapIdentifier)

Description copied from interface: Reduce

Mark the receipt of Output from a given Map instance

Specified by:

setReceivedOutputFrom in interface Reduce

Parameters:

mapIdentifier - The ID of the Map from which an output has been received.

getAwaitingOutputsFrom

public Iterator<String> getAwaitingOutputsFrom()

Description copied from interface: Reduce

Retrieves the set of Map identifiers from which an Output has not been received so far.

Specified by:

getAwaitingOutputsFrom in interface Reduce

Returns:

An iterator of the pending identifiers

resetOutputTracker

public void resetOutputTracker()

Description copied from interface: Reduce

This method resets the output tracker associated with the reducer. This is useful when the stages are part of an iterative computation.

Specified by:

resetOutputTracker in interface Reduce

addReducer

protected void addReducer(Reduce reduce)
                   throws MapReduceException

This method is called on a Map, once it has been added to a Reduce using the Reduce.addMap() method.

Parameters:

reduce -

Throws:

MapReduceException

removeReducer

protected void removeReducer(Reduce reduce)
                      throws MapReduceException

This method is called on a Map, once it has been removed from a Reduce using the Reduce.removeMap() method.

Parameters:

reduce -

Throws:

MapReduceException

addMapper

protected void addMapper(Map map)
                  throws MapReduceException

This method is called on a Reduce, once it has been added to a Map using the Map.addReduce() method.

Parameters:

map -

Throws:

MapReduceException

removeMapper

protected void removeMapper(Map map)
                     throws MapReduceException

This method is called on a Reduce, once it has been removed from a Map using the Map.removeReduce() method.

Parameters:

map -

Throws:

MapReduceException

createResults

public Results createResults(boolean completeResult,
                    boolean processingComplete)

Description copied from interface: MapReduce

Create the Results type to be used for reporting results produced as a result of the processing.

Specified by:

createResults in interface MapReduce

Returns:

The created Results.

writeResults

public void writeResults(Results results)
                  throws MapReduceException

Description copied from interface: MapReduce

Writes the specified results to the default output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

results - The results to be written.

Throws:

MapReduceException - If there are problems in marshalling or writing these results.

writeResults

public void writeResults(ds.funnel.topic.Topic topic,
                Results results)
                  throws MapReduceException

Description copied from interface: MapReduce

Writes the specified results to a specific output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

topic - The topic to write results to

results - The results that needs to be written

Throws:

MapReduceException - If there are problems in marshalling or writing these results.

writeResults

@Deprecated
public void writeResults(String streamSynopsis,
                           Results results)
                  throws MapReduceException

Deprecated.

Description copied from interface: MapReduce

Writes the specified results to a specific output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

streamSynopsis - The synopsis for the stream

results - The results that needs to be written

Throws:

MapReduceException - If there are problems in marshalling or writing these results.

writeResults

@Deprecated
public void writeResults(int synopsisType,
                           String streamSynopsis,
                           Results results)
                  throws MapReduceException

Deprecated.

Description copied from interface: MapReduce

Writes the specified results to a specific output stream.

Specified by:

writeResults in interface MapReduce

Parameters:

synopsisType - The synopsis type for the stream in question

streamSynopsis - The synopsis for the stream

results - The results that needs to be written

Throws:

MapReduceException - If there are problems in marshalling or writing these results.