Analyze Actual Trades (Uses Table of Data Loaded from a Tabular File on Disk) (Point-and-Click NinjaTrader® Strategy)

Trading Strategy Summary

This Quagensia Strategy will help you answer all of the following questions:

1. How many of your losing trades could have been avoided if you could have skipped all of your trades that were made based on trading ideas or trading setups that didn’t work in the past, as determined by a quick backtest of a Quagensia Strategy?

2. Is there a simple filter you could have applied to your criteria to enter trades in the past that would have filtered out many of your losing trades without filtering out too many of your winning trades, thereby improving your overall trading performance?

3. How many of your exits for the trades that you took could have been improved by testing your exit strategies with some quick backtests and optimizations of a Quagensia Strategy that contained your exit logic?

4. How good have the historical recommendations been from a given source such as an analyst, TV personality, market newsletter, trading idea website, X handle, or other social media account?

5. Which analysts or other sources of buy and sell signals have been the best and worst historically, and to what extent do different exit strategies affect the answer to this question?

6. Had you entered a trade based on every historical recommendation from an analyst or other source of buy and sell signals, what exit strategies would have worked best?

This Quagensia Strategy helps you answer the questions above by loading entry signals from a tabular data file on disk into a data table using Quagensia N Edition’s innovative table system, entering positions based on the entry signal times in the file, configurable “entry signal category” and/or custom metric filters, and/or your own custom entry logic that you can create without code, and exiting positions based on a configurable trailing stop, profit target and/or time stop and/or your own custom exit logic that you can create without code.

The entry signals can be those of your actual historical trades downloaded from your broker, they can be entry signals that are recommendations from various analysts, TV personalities, market newsletters, trading idea websites, and X handles or other social media accounts that issue buy and sell recommendations, or they can be entry signals that are generated by any other piece of software you use such as another trading platform, Bloomberg® Terminal®, Refinitiv® Eikon®, MATLAB®, or R.

If you are not in possession of a data file containing entry signals but would still like to test this Quagensia Strategy on an example data file with randomly generated entry signals, set the “If File Does Not Exist, Create an Example File with Random Trades” input parameter to “True”. The data file will be created at the file path to which the “File Path of Entry Signals to Load from Disk” input parameter is set and the backtest or optimization will run using this file. When you get a data file of actual entry signals you can replace this file of randomly generated signals with the file containing the real entry signals you want to analyze.

The file can contain entry signals for all the instruments you trade, the entry signals can be sorted in any order, and the file can have optional columns such as the order quantity, the category name and ID, and a custom metric that can be a decimal number or a whole number. Read NOTE #1 below to learn how to use these columns and format the file, and read all the notes below, the descriptions of this Quagensia Strategy’s input parameters, and the messages in the yellow “comments” below to gain a more complete understanding of this highly configurable Quagensia Strategy.

To customize this Quagensia Strategy without code, scroll down to the two yellow “comments” that start with the following message and follow their instructions:

”
********************************************************************
********************************************************************
Attention paid users
********************************************************************
********************************************************************
”

This Quagensia Strategy demonstrates how to do the following powerful things with tables of data:

1. Load a table of data from a tabular data file on disk into an in-memory data table.

2. Query and manipulate the data using 1) filtering and 2) sorting.

3. Do queries whose results overwrite the data in the queried table of data.

4. Do queries whose results are copied into a new table of data rather than overwriting the queried table of data.

5. Loop over the rows of a table of data and get the values of the fields in each row.

6. Loop over the rows of a table of data and update the values of the fields in each row.

NOTE #1: This Quagensia Strategy requires a tabular data file in a specific format (as described in detail below) to work properly exactly as-is, without any customizations. If you are in possession of such a file you can set the “File Path of Entry Signals to Load from Disk” to the file path of the file containing the entry signals to load from disk. If you are not in possession of such a data file, set the “If File Does Not Exist, Create an Example File with Random Trades” input parameter to “True” and set the “File Path of Entry Signals to Load from Disk” input parameter to the file path where you would like the example file with random trades to be created and then loaded from disk.

Here is an example of a file path:

C:\Folder A\Folder B\Folder C\EntrySignals.csv

The file can be named something different and its file extension doesn’t need to be “.csv”, but a folder with the folder path must exist and the running instance of NinjaTrader® must be allowed to create and modify files in the folder with that folder path. To increase the chances of NinjaTrader® being allowed to create and modify files in the folder with the folder path to which this input parameter is set, you can use the folder path of a folder where you normally save files from other applications. It is recommended to create a subfolder just for reading and writing data files from NinjaTrader® to reduce the risk of overwriting any of your other files.

For this Quagensia Strategy to work properly exactly as-is, without any customizations, the tabular data file must be formatted in the following way:

i. It must have a column header row. This row can be blank or it can have between two and six column header titles.

ii. The file’s first two columns must be the “Entry Signal Time” and “Instrument” in that order. These are the only two columns that are required. Here’s an example of a correctly formatted file with only these two columns:

Entry Signal Time|Instrument
“2019-01-21 13:55:50″|”SPY”
“2022-09-14 22:47:05″|”SPY”

iii. The file supports up to six columns, which must be in this order:

Entry Signal Time
Instrument
Order Quantity (Negative Means Enter Short Not Long) (Optional)
Category (Optional)
Category ID (Optional)
Custom Metric (Optional)

Here’s an example of a correctly formatted file with all six columns. Note that their should be no line break in the header row as there is in the example below. The example file below should have three lines total.

Entry Signal Time|Instrument|Order Quantity (Negative Means Enter Short Not Long) (Optional)|Category (Optional)|Category ID (Optional)|Custom Metric (Optional)
“2015-01-25 17:21:54″|”SPY”|”100″|”Analyst A”|”1″|”6”
“2017-11-20 04:26:31″|”SPY”|”100″|”Analyst B”|”1″|”8”

Your file does not need to have any of the four optional columns at the end of the list above, but if you include an optional column in your file, any optional columns that are before it in the list of six columns above must either have real data in them or must at least have column values that are empty. For instance, if your entry signals have Category values but not Order Quantity values, the file could like like this:

Entry Signal Time|Instrument|Order Quantity (Negative Means Enter Short Not Long) (Optional)|Category (Optional)
“2022-09-14 22:47:05″|”SPY”||”An Analyst’s Buy Recommendations”
“2019-01-21 13:55:50″|”SPY”||”A TV Personality’s Buy Recommendations”

In the example above, note the two field delimiter characters without a value between them. This signifies that the entry signal row does not have an Order Quantity but does have a Category. Note also the lack of field delimiters after the Category column, indicating that this file does not have values for Category ID and Custom Metric.

iv. The tabular data file’s encoding type either has to use the “Unicode (UTF-8)” file encoding like most files do, or you need to expand the section beneath the message that starts with “STEP #2 of 5: Before any other logic is executed by this strategy, do the following one time only (not once per bar)” and change the “File Encoding Type” input parameter of the “Create Tabular Data File Definition” Quagensia Function call (which you will find after scrolling about 70% of the way down this newly expanded section to the first yellow “comment” and looking directly beneath the yellow “comment”) to the file format that your file uses.

v. Values in the “Entry Signal Time” column either need to be in Quagensia Format for date and time data (“yyyy-MM-dd HH:mm:ss.FFFFFFF”) (Examples: 2018-29-05 14:45:58 and 2018-29-05 14:45:58.1234567), or you need to expand the section beneath the message that starts with “STEP #2 of 5: Before any other logic is executed by this strategy, do the following one time only (not once per bar)” and make one of the following sets of changes to the input parameters of the “Create Tabular Data File Definition” Quagensia Function call (which you will find after scrolling about 70% of the way down this newly expanded section to the first yellow “comment” and looking directly beneath the yellow “comment”):

—-v-1. If the date and time data is in a format that is one of the standard formats in your culture or region of the world (and your Microsoft Windows “Regional” settings are set to use this culture or region of the world, as they almost always are) and you will not be sharing your modified version of this Quagensia Strategy with anyone else whose Microsoft Windows “Regional” settings are different than yours, change the “Formatting Rules For Numbers, Dates, and Times” input parameter’s value to “The Personal Formatting Preferences of Whoever Uses This Algo” and set the “Parse Dates and Times Exactly Based On Selected Formats For Date/Time Data” input parameter’s value to “False”.

—-v-2. If the date and time data is in a format that is one of the standard formats in your culture or region of the world or is in one of the standard formats in a culture or region of the world you recognize, change the “Formatting Rules For Numbers, Dates, and Times” input parameter’s value to the appropriate culture or region which can be found in the subfolders that are under the “Select a Different Set of Formatting Preferences” option and set the “Parse Dates and Times Exactly Based On Selected Formats For Date/Time Data” to “False”. There are hundreds of cultures and regions to choose from, but they are grouped alphabetically by spoken language then culture or region of the world so you shouldn’t have a problem finding your specific culture or region of the world as long as your installation of Microsoft Windows has this specific culture or region installed, which it almost certainly will.

—-v-3. If the date and time data is in a specific format that is not one of the standard formats in your culture or region of the world and is not in a standard format in another culture or region of the world you recognize, change the “Formatting Rules For Numbers, Dates, and Times” input parameter’s value to “The Best Formatting Preferences For Sharing Between Users”, set the “Parse Dates and Times Exactly Based On Selected Formats For Date/Time Data” to “True”, and then either change the “Format For ‘Date and Time’ Data” input parameter’s value to the specific format, or if the specific format is not present in the list, add a Quagensia Function call to the Quagensia Function located in the context-sensitive context menu under “More Actions >> Tables and Tabular Data Files >> Tabular Data Files >> Tabular Data Files >> Modify Tabular Data File Definition >> Dates, Times, and Numbers >> Dates and Times >> ‘Date and Time’ Formatting >> Modify Tabular Data File Definition Setting: Set “Format For ‘Date and Time’ Data” To A Custom Format (Advanced)” directly after the “Create Tabular Data File Definition” Quagensia Function call and set the “Format For ‘Date and Time’ Data” input parameter to the specific date and time format. This last way of instructing the Quagensia Strategy how to interpret the date and time field is the hardest to do correctly, but you can learn more about how to use the “Modify Tabular Data File Definition Setting: Set “Format For ‘Date and Time’ Data” To A Custom Format (Advanced)” Quagensia Function just mentioned by opening it up and reading its documentation.

vi. Values in the “Instrument” column must match the symbol used by NinjaTrader® for this instrument. This column is not case-sensitive.

vii. Values in the optional “Order Quantity (Negative Means Enter Short Not Long) (Optional)” column must be positive whole numbers to indicate long entries or negative whole numbers to indicate short entries. The value 0 is not allowed. If the data file does not have a non-empty Order Quantity column or if some of the entry signal rows in the data file have empty Order Quantity fields, the whole number entered into the “Default Order Quantity (If Missing From File) (Negative Means Enter Short Not Long)” input parameter will be used as the order quantity for the entry orders for those entry signal rows.

viii. Values in the optional “Category (Optional)” column can be any text such as the source of the trading idea represented by the entry signal, for example the name of an analyst, TV personality, newsletter, or trading idea website. If the entry signal was your own idea, this field can represent your rationale for taking the trade. This column is not case-sensitive.

ix. Values in the optional “Category ID (Optional)” column can be any whole number other than -1. This column serves the same purpose as the “Category (Optional)” column in that it allows you to categorize entry signals and backtest only entry signals in a certain category, with the added benefit that it is a whole number, which can be optimized over in an optimization in the NinjaTrader® Strategy Analyzer to quickly compare the relative backtest performances of all the categories of entry signals in the tabular data file.

x. Values in the optional “Custom Metric (Optional)” column can be whole numbers or decimal numbers that represent any numeric quantity. For instance, the value could be 5 for a strong buy recommendation, 4 for a buy, 3 for a hold, 2 for a sell, and 1 for a strong sell. This value could represent your confidence in the entry signal or its source, or it could represent the number of analysts who recommend buying the instrument or the ratio of analysts who are bullish or bearish on the instrument. If the data file has a non-empty “Custom Metric (Optional)” column and some of the entry signals’ “Custom Metric (Optional)” fields are greater than or equal to the decimal number to which the “Minimum Custom Metric Threshold” input parameter is set and less than or equal to the decimal number to which the “Maximum Custom Metric Threshold” input parameter is set, only those entry signals will be considered in the backtest. If the data file does not have a non-empty “Custom Metric (Optional)” column or an entry signal’s “Custom Metric (Optional)” value is missing, the values of the such “Custom Metric (Optional)” fields are considered to be 0.

xi. The single character that separates the fields in a single row of data from each other must be the same as the single character in the “Field Delimiter (1 Character Only)” input parameter.

For example, in the following row of data, the pipe (|) character is the “field delimiter”:

“SPY”|”2020-10-02 12:18:34″|100|”My Trades”|1|10

The field delimiter should be a special character that is shared and understood across all countries, cultures, and spoken languages around the world, not a letter of an alphabet or a spoken language-specific character in a spoken language that uses characters rather than letters.

Three popular choices for field delimiter characters are the comma, semicolon, and pipe (|).

xii. The column header titles and fields in the tabular data file need double quotes around them if the value in the column header title or field has a character that is the same as the field delimiter used by the file or if the column header title or field has a double quotes (“) character.

xiii. All double quotes (“) characters in the column header titles and fields in the tabular data file need a second double quotes character before them to “escape” the double quotes character so that it will not be mistaken for the beginning or end of a field’s value.

NOTE #2: When backtesting on 1-day daily bars, the value of the “Entry Signals Occurring After Hours Should Be Entered on Next Open (Daily Bars Only)” input parameter determines at which bar’s open time each entry signal’s entry market order will be submitted and filled given the earliest possible order submission time for an entry signal in the data file, which is the time of the entry signal in the data file plus the number of seconds to which the “Delay Between Entry Signal And Entry Order Submission (In Seconds)” input parameter is set.

When backtesting on 1-day daily bars when the “Entry Signals Occurring After Hours Should Be Entered on Next Open (Daily Bars Only)” input parameter is set to “True”, then if the earliest possible order submission time for an entry signal in the data file occurs during a time when the market is believed to be closed by the backtest engine, such as a time that is after hours or on a weekend, the backtest engine will enter a position at the open time of the bar directly after the after hours or weekend period. For example, if the earliest possible order submission time for an entry signal in the data file occurs on a weekend where the following Monday is not a market holiday, the open time at which the market order will be submitted and filled by the backtest engine will be the open time of the Monday bar. Note that when used in this mode, this Quagensia Strategy needs to be able to determine if a time that occurs after the current bar being evaluated occurs before the next daily bar or not. This is accomplished by using NinjaScript® functions that rely on the data in the NinjaTrader® holiday database, which can be viewed by clicking on the “Tools” main menu item from within NinjaTrader®, selecting the “Trading Hours” menu item to open the Trading Hours window, and selecting the “Trading Hours Template” of interest from the “Templates” list box. When you do this, you may find that the holiday data for a given instrument does not go back in time as far as you would like to backtest your entry signals. If you backtest entry signals whose Entry Signal Time values are earlier than the earliest holiday data for an instrument or later than the latest holiday data for an instrument or if the holiday data is otherwise incomplete or incorrect, this Quagensia Strategy’s determination of whether a time that occurs after the current daily bar being evaluated in a backtest occurs before the next daily bar or not will sometimes be incorrect, in which case the entry market order for the entry will be submitted one bar earlier or later than this Quagensia Strategy would have submitted the entry market order.

When backtesting on 1-day daily bars when the “Entry Signals Occurring After Hours Should Be Entered on Next Open (Daily Bars Only)” input parameter is set to “False”, then if the earliest possible order submission time for an entry signal in the data file occurs during a time when the market is believed to be closed by the backtest engine, such as a time that is after hours or on a weekend, the backtest engine will enter a position at the open time of the bar that is one bar *after* the bar that is directly after the after hours or weekend period, i.e. the entry order will be submitted with an additional delay of one full bar. For example, if the earliest possible order submission time for an entry signal in the data file occurs on a weekend where the following Monday is not a market holiday, the bar open time at which the market order will be submitted and filled by the backtest engine will be the bar open time of the Tuesday bar, not the bar open time of the Monday bar.

Note that backtests on intraday bars and bars other than 1-day daily bars ignore the value of this input parameter and will submit and fill entry market orders using the logic that was just described for 1-day daily bars when the “Entry Signals Occurring After Hours Should Be Entered on Next Open (Daily Bars Only)” input parameter is set to “False”. For example, if the earliest possible order submission time for an entry signal in the data file occurs on a weekend in a backtest on 1-minute bars, the bar open time at which the market order will be submitted and filled by the backtest engine will be the open time of the second 1-minute bar of the new week, not the open time of the first 1-minute bar of the new week. This same delayed fill behavior also occurs when backtesting intraday bars, such as 1-minute bars, when the earliest possible order submission time for an entry signal in the data file occurs during the normal session at a time where there is a gap in the intraday bars such as may occur on 1-minute bar charts of instruments that are so lightly traded that they sometimes don’t have any trades for an entire minute. If an entry time occurs during such a market data gap, the bar open time at which the market order will be submitted and filled by the backtest engine will be the bar open time of the bar *after* the first bar after the market data gap, not the bar open time of the bar directly after the market data gap.

NOTE #3: If the times in the Entry Signal Time column in the data file are not in New York time you will need to set the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter to the Time Zone ID of the Microsoft Windows time zone of the times in the Entry Signal Time column of the data file.

There is an easier way and a harder way to do this.

You can do it the easier way if you haven’t already sent the code to NinjaTrader® with a given value in the “Strategy Name Inside of NinjaTrader ®” field, which is near the top of this strategy’s workspace. To do this, set the default Windows time zone ID to which the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter will be set when you send the code to NinjaTrader® for the first time. This is done by first creating your own customized copy of this built-in strategy, changing its “Strategy Name” and “Strategy Name Inside of NinjaTrader ®” fields, which are near the top of this strategy’s workspace, and saving the copy of this built-in strategy to your “My Algos” folder, then selecting your time zone from the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter’s drop-down list from inside your customized copy of this strategy in Quagensia N Edition. Note: If you have already sent the strategy’s code to NinjaTrader®, you can still do it this way if you change the value in the “Strategy Name Inside of NinjaTrader ®” field near the top of this strategy’s workspace, then select your time zone from the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter’s drop-down list from inside Quagensia N Edition, and send it to NinjaTrader® again with the new value in the “Strategy Name Inside of NinjaTrader ®” field.

You will have to do it the harder way if you have already sent this strategy’s code to NinjaTrader® with a given value in the “Strategy Name Inside of NinjaTrader®” field and don’t want to change the value in that field.

If this is your situation, first set this strategy’s “Print List of Valid Time Zone IDs to NinjaScript® Output Window” input parameter to “True” by checking its checkbox from within NinjaTrader® and run a backtest of this strategy on any instrument with the NinjaScript® Output Window open to get a full list of the time zone display names and Time Zone IDs of the time zones that are available in your installation of Microsoft Windows. Scroll to locate the time zone of the times in the Entry Signal Time column of the data file of entry signals, copy the text next to the text “ID:” without copying any space on the left or the right of the Time Zone ID text, and paste it into the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter. Note that the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter is case-sensitive and no extra spaces are allowed on either side of the Time Zone ID.

How to quickly recover from a mistyped Time Zone ID: If the value in the “Time Zone ID of Entry Signal Times (Case-Sensitive, No Extra Spaces)” input parameter in NinjaTrader® does not match the Time Zone ID of any time zone in your installation of Microsoft Windows and you attempt to run a backtest, an error will occur. The full list of time zones and their IDs, as described above, will be printed to the NinjaScript® Output Window as long as it was open during the backtest, even if you do not set the “Print List of Valid Time Zone IDs to NinjaScript® Output Window” input parameter to “True”. As long as you follow the instructions above regarding carefully copying and pasting the Time Zone ID and are sure not to copy any spaces on the left or the right of the Time Zone ID, this error will not occur.

NOTE #4: This strategy uses Quagensia N Edition’s Quagensia Table System, which is also called the “tables of data” feature. This feature only works properly if the proper versions of the QuagensiaNEditionRuntime.dll, QuagensiaNEditionRuntime.cs, and QuagensiaRuntime.dll files are present in the NinjaTrader® Bin-Custom folder and the QuagensiaNEditionRuntime.dll and QuagensiaRuntime.dll files are properly “referenced”, all of which is accomplished by importing the QuagensiaNEditionRuntime.zip NinjaScript® add-on file that comes with Quagensia N Edition into NinjaTrader®. If the proper versions of the QuagensiaNEditionRuntime.dll, QuagensiaNEditionRuntime.cs, and QuagensiaRuntime.dll files are not present in the NinjaTrader® Bin-Custom folder when you generate the code for a Quagensia N Edition Strategy that uses the “tables of data” feature, the Quagensia N Edition Runtime Setup Assistant will automatically open and will guide you through the quick and easy process of importing the QuagensiaNEditionRuntime.zip NinjaScript® add-on file that comes with Quagensia N Edition into NinjaTrader®.

NOTE #5: This strategy supports holding multiple positions for the same instrument at once, as long as they are all long or all short, to make it possible to run this strategy on a tabular data file with lots of entry signals that occur at or close to the same time, and whose positions are not exited before the next entry signal. For this feature to work properly, the “Entries per direction” in the NinjaTrader® Strategy Analyzer must be set to a number that is no smaller than the largest number of concurrent long or short positions. The “Entries Per Direction” in the “Strategy Details” section below has been set to 1000000 to set the default value of the “Entries per direction” in the NinjaTrader® Strategy Analyzer to a sufficiently large number.

NOTE #6: While this strategy supports holding multiple positions for the same instrument at once, concurrent positions must be either all long or all short. If the entry signals in the tabular data file are a mix of long and short entry signals, the entry market order for any long position will first close out all currently open short positions and then go long, and the entry market order for any short position will first close out all currently open long positions and then go short.

NOTE #7: This strategy makes the simplifying assumption that the backtest engine will exit the entire position for a given entry position. This assumption is not guaranteed to hold up in live trading, where a limit order to exit a position could only get partially filled in certain scenarios.

NOTE #8: If you will only be running this strategy as-is, i.e. if you will not be adding your own custom entry filters or custom exit filters in the “When Bar Updates” section below, the “Bars required to trade” field in the NinjaTrader® user interface can be as low as 1, since by default this report considers only the current price bar and the price bar directly before it. If you will be running this report with your own custom entry filters or custom exit filters, the “Bars required to trade” property will likely have to be larger to account for things like a 10-period simple moving average needing at least 10 price bars of initial data before it can provide a valid result for a given price bar. The larger you make this number, the less likely you will experience run-time errors or logic errors, but the more initial price bars will be required before the first price bar will be considered for inclusion in this report. If you use a really big number for the “Bars required to trade” property, like 200, you will notice that the “Start Date of Backtest” displayed on the chart and in the NinjaTrader® Output window will be the bar end time for the price bar approximately 200 bars after the first price bar occurring on or after the “Start date” that you set in the NinjaTrader® Strategy Analyzer, because in the example just given, the first 200 trading bars of data would be skipped before the logic in the “When Bar Updates” section below would be executed for each price bar in the backtest.

Quagensia N Edition’s innovative Quagensia Table System makes it easy to load tables of many kinds of data, not just time series data, from tabular data files into in-memory data tables that you can query, modify, and then write back to tabular data files on disk from your strategies.

If you have valuable data in Microsoft Excel files, a database such as Microsoft SQL Server, SQLite, or MySQL, if you have access to exotic data via Bloomberg® Terminal®, Refinitiv® Eikon®, or another high end institutional platform, if you have tabular data output from MATLAB® or R, or if you have access to alternative data sources on the internet, it is generally very easy to get these kinds of data into comma-delimited, semicolon-delimited, or otherwise character-delimited tabular data files. And once you do, it is easy to get all of this data, even entire databases of data, into your NinjaTrader® strategies by simply clicking choices on the screen.

You can also create in-memory data tables and populate them with data from scratch, i.e. not from tabular data files, and then create, modify, and delete rows from these in-memory data tables.

Regardless of how you create your in-memory data tables, Quagensia N Edition makes it easy to write that data to tabular data files on disk.

All features just described work with data columns of 73 different types of data including: 10 numeric data types, 3 date/time data types, colors, Booleans (true or false), text, and 57 “lists of choices” data types such as the months of the year or the days of the week.

Copyright © 2023 Quagensia, Inc. All rights reserved.