QDS/dxFeed API 3.110 has incorporated 82 changes since previous public release 3.102. Important changes to dxFeed API since 3.102 are: * [QD-186] dxFeed API: add isSpreadLeg and isExtendedTradingHours flags to TimeAndSale event. * [MDD-261] Add price type flags to Summary record. * [QD-129] dxFeed API: Wildcard subscription to events - com.dxfeed.api.osub.WildcardSymbol class is introduced to represent "wildcard" subscription symbol. - All event types support wildcard subscription with a limitation that lasting events events are not conflated when subscribed to via wildcard and wildcard subscription should not be mixed with regular by-symbol subscription in a single instance of DXFeedSubscription. * [QD-130] dxFeed API: Ability to publish events via stream - All events are published to stream in addition to their regular contracts. - Autogenerated xxxEventDelegateFactory classes now clearly show what contracts are used for each record. * [QD-120] dxFeed API: Message and Configuration events (subscription and publishing) - com.dxfeed.event.misc.Message event added; it uses stream contract - com.dxfeed.event.misc.Configuration event added; uses ticker contract and supports version for ordered updates * [QD-192] dxFeed API: Introduce CandlePrice.SETTLEMENT * [QD-188] dxFeed API: Shorten toString representation of market events to make logs more compact and easy to read. * [QD-140] dxFeed API: Misc performance optimizations - List of event and subscription listeners in DXFeedSubscription class is optimized for a common case of a single listener. * [QD-177] dxFeed API: toString() method of Summary event prints DayClosePriceType instead of PrevDayClosePriceType. * [QD-172] dxFeed API: incorrect sign extension in created TimeAndSale events * DXFeed API: JavaDoc for Order is clarified with respect to SIDE_BUY and SIDE_SELL. IMPORTANT NOTE REGARDING UPGRADING TO DXFEED API 3.1xx FROM EARLIER VERSIONS: INCOMPATIBLE CHANGES: * DXFeed.createConnectedFeed method is removed. Use DXEndpoint to create connection. See DXFeed class javadoc for details. * "com.dxfeed.api.event" package is renamed to "com.dxfeed.event.market". Package contents did not change (the same Quote, Trade, TimeAndSale, etc classes are there). DXFeedSubscriptionChangeListener is moved to another package and renamed to ObservableSubscriptionChangeListener. This change should not affect feed consumers. NEW FEATURES: * "com.dxfeed.event.candle" package is introduced with a set of CandleXXX classes to receive chart candles. Candle class represents one Open-High-Low-Close-Volume candle with auxiliary information. Use instances of CandleSymbol class to specify which Candles to subscribe to. * "DXFeedTimeSeriesSubscription" class is introduced to subscribe to time series (history) of past events in addition to all new/current events. It is designed to be used with TimeAndSale and Candle events. * "DXFeed.createTimeSeriesSubscription" method is introduced. LIST OF ALL CHANGES SINCE 3.102: QDS 3.110: * [QD-192] dxFeed API: Introduce CandlePrice.SETTLEMENT * [QD-189] Misc improvements for xxxMapping classes and link records with mappings - Provide DataRecord.getMapping() method that returns the mapping for the corresponding record instead of xxxMapping.getInstance(). - Introduce com.devexperts.qd.ng.RecordMapping as a base class for all xxxMapping classes. - Provide xxxMapping.getRecord() method for all mappings. - Provide xxxMapping.getRecordExchange() methods for market event mappings. - Provide xxxMapping.getRecordPeriod() and getRecordPrice() for candle mappings. * Better handling of decimal fields in mapping classes. * Add composite and regional Book records and mappings without event class. * Sort records in the scheme so that all regional records go after all composite records. * Add method MarketEventSymbols.buildSpreadSymbol() for building canonical spread symbols. QDS 3.109: * [QD-188] dxFeed API: Shorten toString representation of market events to make logs more compact and easy to read. * [QD-186] dxFeed API: add isSpreadLeg and isExtendedTradingHours flags to TimeAndSale event. * [QD-185] dxFeed Scheme: Add "Candle{=m,price=xxx}" records for xxx in bid, ask, mark * [QD-183] dxlib: More util classes (QuickSort and LongHashXXX classes) * [QD-182] Schedule shall provide more details when it fails to parse schedule definition. * [QD-152] dxlib: Logging needs to implement company-standard log layout * [QD-129] dxFeed API: Wildcard subscription to events - com.dxfeed.api.osub.WildcardSymbol class is introduced to represent "wildcard" subscription symbol. - All event types support wildcard subscription with a limitation that lasting events events are not conflated when subscribed to via wildcard and wildcard subscription should not be mixed with regular by-symbol subscription in a single instance of DXFeedSubscription. * [QD-98] Drop support for legacy (pre QDS 3.69) implementations of QTP protocol that work in data-provider role - QDS distributor adapter waits for PROTOCOL_DESCRIPTOR message. - Subscription will be sent to remote host only when the appropriate protocol descriptor is received. * [QD-112] Build-in dxFeed filters * Add QuickSort class. * [QD-140] dxFeed API: Misc performance optimizations - List of event and subscription listeners in DXFeedSubscription class is optimized for a common case of a single listener. * [QD-130] dxFeed API: Ability to publish events via stream - All events are published to stream in addition to their regular contracts. - Autogenerated xxxEventDelegateFactory classes now clearly show what contracts are used for each record. * [QD-76] Extend default dxFeed scheme to eliminate the need to use toscommon scheme * Add missing fields in Trade and Profile records. * Add math methods to Decimal, improve performance. * [QD-167] dxFeed API: Provide **Mapping classes for all records in impl package to be used by exchange gateways - Mapping API is semi-public (to be used only by exchange gateways and similar utilities). - Mapping classes are located in com.dxfeed.event..impl package. - The general name of the mapping class is Mapping. - Each mapping class has getInstance method to get a mapping for a DataRecord. - Each mapping class has getter and setter methods for all its record field. - Some fields have multiple getter and setter types distinguished by suffixes. For example, time fields have "getXXXSeconds" and "getXXXMillis" methods as well as the corresponding setters. - dxFeed event delegate classes use mapping classes in their implementations. - Main entry point for code generation tool is now "com.dxfeed.api.codegen.ImplCodeGen" - To define mappings and delegates for new event types just edit the source of ImplCodeGen. It creates all the corresponding files on the first invocation using its build-in templates. - Source files that don't need any changes are not touched by ImplCodeGen. ImplCodeGen can be invoked at any time to verify that all auto-generated sources are up-to-date. - Single CandleMapping class is used for all Trade. records as well as for Candle record. * [QD-81] QDS Tools: Ability to shield comma in qds connect symbol list - Comma in a symbol can be now shielded with back-slash. - Literal back-shall in the symbol has to use double back-slash. - Comma inside any kind of parentheses (straight, brace, or curly) is automatically shielded to account for a common use case of addition attributes that follow symbol in parentheses. Parenthesis themselves can be shielded with back-slash to avoid such effect. * [QD-178] QDS splitted tapes may start with payload data written prior to DESCRIBE RECORD data. - ByteArrayParser requires DESCRIBE_RECORD messages if instructed to use DESCRIBE_RECORD. Messages are considered corrupted if they refer to undescribed records. - MessageAdapter can skip corrupted and unknown messages if the following system properties are set: -Dcom.devexperts.qd.qtp.MessageAdapter.skipCorruptedMessages -Dcom.devexperts.qd.qtp.MessageAdapter.skipUnknownMessages * [QD-177] dxFeed API: toString() method of Summary event prints DayClosePriceType instead of PrevDayClosePriceType. QDS 3.108: * [QD-174] [Instruments] Add ability to compare IPF date-based fields to current date * [QD-171] QDS Tools: Kill JVM when attempt to initialize any tool fails * [QD-175] Make IndexedSet and IndexedMap cloneable. * QD RMI: Fix data race that was found by data race detector and a potential bug with detection of least-loaded connection that could materialize if >4GB buffered to send (bytes were counted in int). QDS 3.107: * [QD-160] QD Core: Discrepancy between QDDistributor.close java doc ¬-nd actual behavior - Documentation of QDDistributor.close is updated. The behaviour of various method after the distributor is closed is unspecified and is left up to implementation. * [QD-164] Remove QDStats from Connectors JMX nodes - MessageConnectorMBean interface is extracted from MessageConnector interface. It defines only properties and operations that every message connector exposes via JMX. - Stats attribute is no longer exposed. * [QD-165] QTP Protocol documentation has wrong information about time and date serial types - DESCRIBE_RECORDS documentation fixed. * [QD-172] dxFeed API: incorrect sign extension in created TimeAndSale events * [QD-170] [IPF] Improve 'instruments' tool to be able to subtract one ipf from another - New option "exclude" was added. It can be specified several times to exclude profiles from several sources. Option "exclude" reads IPF files similar to "read" option using same parameters specified via "bizdate" and "osi" options. Option "exclude" is executed after "merge" option and before "check" option. - New option "script" was added. It can be specified several times to execute several scripts. Option "script" executes commands from specified script file one-by-one. Specified script file shall contain commands, empty lines and comments marked with hash symbol at the start of the line. Each command occupies single line in the script, starts with command name and ends with command arguments. Available commands include all available options (full names only) plus command "clear" which clears working list of profiles. Command "script" allows recursive execution of other scripts. Commands "read" and "exclude" use "bizdate" and "osi" parameters specified as command-line options - there is no way to change their value from within the script. Option "script" is executed after "write" option. - The order of option execution is same as before, it can be checked from built-in help, it does not depend on the order in which options are specified, and all multi-specified options are executed together in their due order. The order of script command execution is according to their order in the script, they are executed strictly one-by-one. All scripts specified by option "script" are executed after all other options, they inherit and share same list of profiles. However it is recommended to avoid mixing old-style options with "script" option to avoid confusion. - Sample script: # Extract options into separate IPF read all.ipf transform if (TYPE != 'OPTION') delete(); sort write options.ipf clear # Write other instruments into other.ipf read all.ipf exclude option.ipf sort write other.ipf * [QD-120] dxFeed API: Message and Configuration events (subscription and publishing) - com.dxfeed.event.misc.Message event added; it uses stream contract - com.dxfeed.event.misc.Configuration event added; uses ticker contract and supports version for ordered updates * [QD-173] RMI: Ability to share socket between RMI and dxFeed API - RMIEndpoint.create(DXEndpoint.Role) method added; it supports FEED or PUBLISHER roles - RMIEndpoint.getDXEndpoint() method added; it returns associated DXEndpoint which shares socket with RMI QDS 3.106: * [QD-161] dxFeed API: Candles are published to wrong (all) periods - Bug fixed, unit-test added. * [MDD-261] Add price type flags to Summary record. * [QD-158] dxlib: Optimize the size of com.devexperts.logging * [QD-149] dxlib: Introduce SynchronizedIndexedSet - [Synchronized]IndexedSet.putIfAbsentAndGet method is introduced. * DXFeed API: JavaDoc for Order is clarified with respect to SIDE_BUY and SIDE_SELL. * [QD-156] FileConnector: "delayed=xxx" or "start=-xxx" starts only at the beginning of a new file - File reader now correctly identifies the first file that contains requested start time. It initially creates a list of all files and looks at the first one whose timestamp does not exceed start time. * [QD-155] FileConnector: Invalid delay is reported to MARS and delay creep when using "start" attribute - "delay" node in mars is renamed to "delayed" to match the name of configuration property "delayed" whose value it reflects. - "delayed" node in mars is "null" when "delayed" is not explicitly set and is formatted as time period (like PT20M) to avoid confusion with "delay_actual". - When using "start=xxx", the effective delay is computed immediately at the start of the file reader (using system current time) and is only recomputed in "cycle" mode on each subsequent cycle. - When neither "start=xxx" nor "delayed=xxx" is specified, then effective delay is computed based on the actual time of the first record in the first file. Before this file appears, "delay_actual" is reported to MARS as "0". QDS 3.105 * [QD-145] FileAnalysis: Analyze frequency distribution of ascii chars in symbols and build huffman prefix code - "-s|--symbols" options in FileAnalysis tool turns on character frequency stats gathering. - For each individual symbol category (futopt, option, futures, other -- is determined by the symbol prefix) stats are gathered separately, as well as total stats for all ASCII symbols. - Hypothetical number of bytes that an optimal code will consume is shown in comparison to the original code, assuming that a 3-bit sequence is used to signal new code type (analysis of complete code replacement approach without extra bits is also shown for comparison). - Length of encoding is limited to 10 bits by default. The limit can be changed with system property: -Dcom.devexperts.qd.tool.analysis.Symbols.L=xxx * [QD-144] Util: A separate build file to create .jar and javadoc for non-QD-specific utility classes (dxlib) - Top-level javadocs (package and class) for com.devexperts.{io|services|logging|util} are cleaned up. - A list of source files that will be tentatively included into dxlib is created with a separate build file. - Removed dependency on com.devexperts.qd.util.TimeFormat. * [QD-143] Tools: Don't create mars nodes if "-s xxx" was not set - The "-s" ("--stat") option specifies an interval to report statistic to the console and to MARS. MARS should not be initialized (and MARS nodes should not be created) in Tools if "-s" option is not specified. * [QD-142] FileReader: Support CSV (comma separated values) text format - CSV format can be specified in file reader with "format=cvs" property. - CSV format can be specified in tape file writer (dump & connect tools) with "format=csv" property. - CSV format is auto-detected by "##QD_" prefix of the file if not explicitly set. - Tab-separated ("text") format is auto-detected by "==QD__" prefix of the file if not explicitly set. - Default CSV format of onDemand data extractor servlet can be parsed automatically (file reader code is aligned with it). * [QD-141] HTTP Connector: Support reading of files, including onDemand service results - HTTP connector supports "file" property to read a specified http URL with FileReader using GET method. - HTTP connector falls back to file reader automatically if POST method is not supported by the server. So, for example, onDemand data can be read using the following address: http://demo.dxfeed.com:7070/onDemand/data?categories=E-Q&symbols=ACN&start=20100506-1447&end=20100506-1449[user=demo,password=demo,file] - HTTP handler thread now has the same name as the name of its connector. - FileConnector also correctly opens http urls. To explicitly invoke file connector for HTTP use "file:http://..." address. However, user and password cannot be specified with FileConnector. - Note, that file reader does not attempt to read time file if the specified URL has query parameters (like onDemand HTTP in given above). However, without query parameters it works just like with a regular file and tries to read a corresponding ".time" file. * [QD-72] Tape file reader shall be more robust and retry attempts to open files - Automatically detect time file presence on the first file in sequence and require presence of time file (or ignore it) for all subsequent files. - Correctly handle the sudden disappearance (inability to open) either data or time file. Rescan directory and wait while directory is non-empty. - File reader supports "time" parameter to explicitly turn on/off time reading. - "time=none" has the same behavior as "ignoreTime". - "time=text" explicitly turns on timestamp file reading. Data files with missing time will not be read by accident. File reader will only read data files when the corresponding ".time" file is present. - Unspecified "time" parameter is auto-detected based on the first file. - File connector reports its CONNECTING/CONNECTED/DISCONNECTED status properly. - File connector can be stopped/restarted while reading or waiting. - File reader thread now has the same name as the name of its connector. - File connector code is included into "qds.jar" delivery (it was in "qds-tools.jar" only). - File connector reports its configured "delay" in MARS node and the "delay_actual" that show the actual delay. It can be tracked by monitoring tools to detect the case where it cannot keep up with reading files fast enough. - Micro-framework for monitoring of connector-specific attributes via MARS is introduced. com.devexperts.qd.monitoring.Monitored annotation marks connector getXXX methods whose values are reported to MARS by ConnectorsMonitoringTask. Note, that address, connection count and number of bytes read/written per second are always reported for each connector. * [QD-34] Cleanup of written tapes does not delete old files as expected. - Take into account both data and time files when computing consumed disk space. - Time files are taken into account only when time files are being written. - Correctly delete files when file name contains directory name. - Require that "split=xxx" is specified when file name contains '~' character. - Create files only when data is actually needs to be written (do not create empty files). - Support timestamp format (10m, 30s, etc) in "storageTime=xxx". - Automatically create a directly for the tape file - Correctly handle a case when the directory if empty or absent * [QD-118] QDS Tools: --tape has misc problems with timestamps that are written to .time files - Write milliseconds to text timestamp files. - Timestamp files should not be created in split mode if "time=none" is explicitly set. - Timestamp files should be written only if "time=text" or "time=long" is explicitly or implicitly set. "time=text" is set implicitly if file name ends with ".data" or if "split=xxx" mode is used. - Timestamps and data are now written to the correct file during file rollover. Both data and timestamp (with zero position) are written to the freshly opened file. - Freshly opened file is correctly named with current time when there was no data for a while. * Zero date-time is parsed from "0" string by com.devexperts.qd.util.TimeFormat. QDS 3.104 * [QD-119] AbstractDataXXXField classes should be public. * [QD-92] Support cycle playback and stop time in FileConnector. Misc bugs related to reading of timestaped files are also fixed. FileConnector reconfigures and restarts itself on any property change via JMX. * [QD-116] Mics TimeFormat bugs and limitations fixed: - TimeFormat fails to correctly parse date-time without explicit time-zone specific in day-light savings. - TimeFormat fails to parse date-time with milliseconds (but supports formatting of such values). - TimeFormat does not support formatting with milliseconds and timezone. - TimeFormat internally throws exceptions during parsing of valid strings. * Timesync publishes more data to MARS. * More detailed javadoc on DXEndpoint.connect method. * DXFeed API: Fix handling of negative times (incorrect sign extension). QDS 3.103 * Stop using negative index in Order event. * Make nice report for cmeproduct transform error for bad data * [QD-96] Support custom fields in InstrumentProfile * Add SPREAD type to InstrumentProfile * ByteArrayParser minor improvements: - Magic constant 10000 is now configurable via system property com.devexperts.qd.qtp.processThreshold - IndexedSet of partitioned messages is allocated only when needed. * [QD-110] QDS Tools: Support --tape option in dump tool. With this option, dump tool becomes a universal file-conversion tool. It can take file in one format and convert it to anther one. Just use: qds dump -t [format=xxx] --tape option implies --quiet and suppresses output to the console. * [QD-109] QDS FileConnector shall become "disconnected" when file is over. Tools quit on it. The following tools can read data from the files via FileConnector: compare, connect, dump, feed, multiplexor. When taking data from a file they should quit when file is over. * [QD-108] QDS Tools shall use default data scheme. Scheme can be overridden by -Dscheme=xxx system property and this code is integrated into QDFactory.getDefaultScheme() now. Scheme can also be overridden by -Dcom.devexperts.qd.DataScheme=xxx system property where xxx supports the same syntax as -Dscheme=xxx. That is, xxx can be of the form ! where both jar file and class name are optional. Support for this syntax is added to com.devexperts.services.Services class. * QDS version checks and reports mismatch between QDS and QDS Tools. Mismatch between versions is included into QDS version string. * Data scheme class name is logged on the first invocation of QDFactory.getDefaultScheme() * Complete dxFeed API JavaDoc. * DefaultScheme and DefaultRecord now explicitly work only with fields that are derived from AbstractDataIntField and AbstractDataObjField classes. DefaultScheme now explicitly works only with records that are derived from DefaultRecord. * DefaultScheme.setParentReferences, DefaultScheme.verifyScheme, and DefaultRecord.setParentReferences static methods are deprecated. Just remove their usage. All the work the were doing is now performed in the corresponding constructors. * Deprecated methods DataScheme.writeRecordIdentity and readRecordIdentity are removed. * Methods in DefaultScheme and DefaultRecord that are not designed to be overrideable are marked as final. * Introduced FileAnalysis tool to analyze file contents and compression algos * FileReader now supports "readAs" property for all formats including text and blob (use it with "qds connect file:"). Note that binary, binary_delta, and binary_plain are all supported as "format" attribute values for file reader, but they do not produce any difference in behaviour. The actual use of delta compression during reading is detected by bits in DESCRIBE_RECORD messages. * TimeField, SequenceField and DateField classes in com.devexperts.qd.kit are introduced. Presentation flag is SerialFieldType marks the time, sequence, and date fields. * [QD-106] Introduced SerialFieldType.withName and SerialFieldType.forNamedField to define more specific named types for specific fields (like BID_PRICE) that will be associated with advanced compressed methods in the future. * Pretty formatting of Time, Sequence, and Date field values in dxFeed data scheme. * RecordBuffer.replaceRecordAt and replaceSymbolAt methods are introduced to perform in-place modification of record and symbol information in the buffer. * Do not print exception stack trace if remote filter name cannot be parsed (just log a warning).