public abstract class MapReduceBase extends OperationBase implements Map, Reduce
Map
and Reduce
interface.Operation.OperationType
Modifier and Type | Field and Description |
---|---|
protected ds.funnel.topic.Topic |
defaultOutputStream |
protected StreamDataset |
mapReduceDataset |
static String |
MODULE |
Constructor and Description |
---|
MapReduceBase() |
Modifier and Type | Method and Description |
---|---|
int |
addMap(Map map)
Adds a Map that will be reduced.
|
protected void |
addMapper(Map map)
This method is called on a Reduce, once it has been added to a
Map using the Map.addReduce() method. |
int |
addReduce(Reduce reduce)
Adds a Reduce for this particular Map.
|
protected void |
addReducer(Reduce reduce)
This method is called on a Map, once it has been added to a
Reduce using the Reduce.addMap() method. |
Results |
createResults(boolean completeResult,
boolean processingComplete)
Create the
Results type to be used for reporting results
produced as a result of the processing. |
Iterator<String> |
getAwaitingOutputsFrom()
Retrieves the set of Map identifiers from which an Output has not been
received so far.
|
DatasetCollection |
getDatasetCollection()
Retrieve the dataset that is associated with this particular operation.
|
Enumeration<String> |
getMapIdentifiers()
Retrieves the set of Map identifiers
|
protected StreamDataset |
getMapReduceDataset() |
int |
getNumberOfMaps()
Retrieves the number of registered Map instances
|
int |
getNumberOfReducers()
Retrieves the number of registered Reduce instances
|
Enumeration<String> |
getReduceIdentifiers()
Retrieves the set of Reduce identifiers
|
boolean |
isAwaitingOutputs()
Check to see is the Reducer is missing outputs from any of the registered
Maps.
|
void |
marshall(ds.funnel.data.format.FormatWriter mFormat) |
int |
removeMap(String mapIdentifier)
Removes a Map from the list of Maps that are registered to this reducer.
|
protected void |
removeMapper(Map map)
This method is called on a Reduce, once it has been removed from a
Map using the Map.removeReduce() method. |
int |
removeReduce(String reduceIdentifier)
Removes a given reducer from the list of Reducers for a given Map
function.
|
protected void |
removeReducer(Reduce reduce)
This method is called on a Map, once it has been removed from a
Reduce using the Reduce.removeMap() method. |
void |
resetOutputTracker()
This method resets the output tracker associated with the reducer.
|
void |
setReceivedOutputFrom(String mapIdentifier)
Mark the receipt of Output from a given Map instance
|
void |
unmarshall(ds.funnel.data.format.FormatReader mFormat)
Unmarshall the operation from a set of marshalled bytes.
|
void |
writeResults(int synopsisType,
String streamSynopsis,
Results results)
Deprecated.
|
void |
writeResults(Results results)
Writes the specified results to the default output stream.
|
void |
writeResults(String streamSynopsis,
Results results)
Deprecated.
|
void |
writeResults(ds.funnel.topic.Topic topic,
Results results)
Writes the specified results to a specific output stream.
|
changeToExecuteFixedNumberOfTimes, changeToExecuteOnce, changeToExecutePeriodically, changeToExecuteWhenDataAvailable, changeToStayAlive, dispose, execute, getDomain, getExecutionProfile, getInstanceIdentifier, getOperationIdentifier, getProcessingDirectives, getVersionInformation, hasDatasetCollection, hasExecutionProfile, hasProcessingDirectives, initialize, isInitialized, markInitializationAsComplete, needsInitialization, setAsExecuteContinuously, setAsExecuteFixedNumberOfTimes, setAsExecuteOnce, setAsExecutePeriodically, setAsExecuteWhenDataAvailable, setAsStayAlive, setDatasetCollection, setDomain, setInitializationComplete, setInitializationNeeded, setInstanceIdentifier, setOperationIdentifier, setProcessingDirectives, setTeminationConditionReached, setVersionInformation, terminationConditionReached, toString
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
dispose, execute, getDomain, getExecutionProfile, getInstanceIdentifier, getOperationIdentifier, getProcessingDirectives, getVersionInformation, hasDatasetCollection, hasExecutionProfile, hasProcessingDirectives, initialize, isInitialized, needsInitialization, terminationConditionReached, toString
public static final String MODULE
protected StreamDataset mapReduceDataset
protected ds.funnel.topic.Topic defaultOutputStream
public void unmarshall(ds.funnel.data.format.FormatReader mFormat) throws ds.funnel.data.format.FormatException
Operation
unmarshall
in interface Operation
unmarshall
in class OperationBase
mFormat
- The byte[] representation of the operation.ds.funnel.data.format.FormatException
public void marshall(ds.funnel.data.format.FormatWriter mFormat) throws ds.funnel.data.format.FormatException
marshall
in interface ds.funnel.data.format.FormatMarshaller
marshall
in class OperationBase
ds.funnel.data.format.FormatException
public DatasetCollection getDatasetCollection()
Operation
getDatasetCollection
in interface Operation
getDatasetCollection
in class OperationBase
protected StreamDataset getMapReduceDataset()
public int addReduce(Reduce reduce) throws MapReduceException
Map
addReduce
in interface Map
reduce
- The Reducer that will be added to this MapMapReduceException
- If there are problems in adding this Reduce
public int removeReduce(String reduceIdentifier) throws MapReduceException
Map
removeReduce
in interface Map
reduceIdentifier
- The identifier of the Reduce to be removedMapReduceException
- If there are problems during removal.public int getNumberOfReducers()
Map
getNumberOfReducers
in interface Map
public Enumeration<String> getReduceIdentifiers()
Map
getReduceIdentifiers
in interface Map
public int addMap(Map map) throws MapReduceException
Reduce
addMap
in interface Reduce
map
- Add a Map instance for this reducerMapReduceException
public int removeMap(String mapIdentifier) throws MapReduceException
Reduce
removeMap
in interface Reduce
mapIdentifier
- The identifier of the Map to removedMapReduceException
- If there are problems removing the Map.public Enumeration<String> getMapIdentifiers()
Reduce
getMapIdentifiers
in interface Reduce
public int getNumberOfMaps()
Reduce
getNumberOfMaps
in interface Reduce
Map
spublic boolean isAwaitingOutputs()
Reduce
isAwaitingOutputs
in interface Reduce
true
is the Reducer is still awaiting outputs;
false
otherwise.public void setReceivedOutputFrom(String mapIdentifier)
Reduce
setReceivedOutputFrom
in interface Reduce
mapIdentifier
- The ID of the Map from which an output has been received.public Iterator<String> getAwaitingOutputsFrom()
Reduce
getAwaitingOutputsFrom
in interface Reduce
public void resetOutputTracker()
Reduce
resetOutputTracker
in interface Reduce
protected void addReducer(Reduce reduce) throws MapReduceException
Reduce
using the Reduce.addMap() method.reduce
- MapReduceException
protected void removeReducer(Reduce reduce) throws MapReduceException
Reduce
using the Reduce.removeMap() method.reduce
- MapReduceException
protected void addMapper(Map map) throws MapReduceException
Map
using the Map.addReduce() method.map
- MapReduceException
protected void removeMapper(Map map) throws MapReduceException
Map
using the Map.removeReduce() method.map
- MapReduceException
public Results createResults(boolean completeResult, boolean processingComplete)
MapReduce
Results
type to be used for reporting results
produced as a result of the processing.createResults
in interface MapReduce
Results
.public void writeResults(Results results) throws MapReduceException
MapReduce
writeResults
in interface MapReduce
results
- The results to be written.MapReduceException
- If there are problems in marshalling or writing these results.public void writeResults(ds.funnel.topic.Topic topic, Results results) throws MapReduceException
MapReduce
writeResults
in interface MapReduce
topic
- The topic to write results toresults
- The results that needs to be writtenMapReduceException
- If there are problems in marshalling or writing these results.@Deprecated public void writeResults(String streamSynopsis, Results results) throws MapReduceException
MapReduce
writeResults
in interface MapReduce
streamSynopsis
- The synopsis for the streamresults
- The results that needs to be writtenMapReduceException
- If there are problems in marshalling or writing these results.@Deprecated public void writeResults(int synopsisType, String streamSynopsis, Results results) throws MapReduceException
MapReduce
writeResults
in interface MapReduce
synopsisType
- The synopsis type for the stream in questionstreamSynopsis
- The synopsis for the streamresults
- The results that needs to be writtenMapReduceException
- If there are problems in marshalling or writing these results.Copyright © 2015. All rights reserved.