Jump to content

D's: First Look USA Tournament Faction Effect


D_acolyte

Recommended Posts

My father and I got into a conversation about which faction are the strongest in a tournament. He thought it was still neverborn I thought it was gremlins. To address this I decided to look at data from tournaments, there are 2 things I am looking at. The first is simply distribution of faction and the second is there as sign that one or more factions will place more often in the top 3 of a tournament. The theory is what do people tend to think will do in tournaments vs what actually is doing well in the tournaments.

My inclusion criteria is the tournament must have the year date in it or have 20 or more participants and no missing data on http://usa.malifaux-rankings.com/#/events as of November 30. Data on the name of the players was discarded. I am trying to get annual tournaments or tournament that are not just there gaming club. Of this I have 27 tournaments in USA and 455 people.

Table 1.1 Table of Frequency of Faction

Faction

Frequency

Percent

Arcanist

84

18.46

Guild

62

13.63

Gremlins

59

12.97

Neverborn

77

16.92

Outcast

56

12.31

Resurrectionist

67

14.73

Ten Thunders

50

10.99

This shows that the most played factions are the Arcanist fallowed by the Neverborn. If all factions are played evenly then we should see about 65 for each faction. Where numbers like the amount of Resurectionist and guild is probably indicative of this the fact that Arcanists and Neverborn are over 10 higher than 65 and ten thunders is over 10 lower than 65 shows that there may be a belief in how these faction perform in the competitive seen.

Next I will be looking at which factions tend to get placed in the top 3.

Table 2.1: Table of Frequency of Faction in the Top 3

Faction

Frequency

Percent

Arcanist

15

0.185185

Guild

9

0.111111

Gremlins

16

0.197531

Neverborn

5

0.061728

Outcast

13

0.160494

Resurrectionist

14

0.17284

Ten Thunders

9

0.111111

Raw numbers it looks like out of 81 placements we should see about 11 to 12 places per a faction. This looks like gremlins may be the strongest. Raw numbers are only part of the story.

 

Table 3.1: Table of Frequency of Percent in the Top 3 Based on their Faction Occurrence.

Faction

Frequency of Occurrence

Frequency in Top 3

Percent in Top 3

Arcanist

84

15

0.178571429

Guild

62

9

0.14516129

Gremlins

59

16

0.271186441

Neverborn

77

5

0.064935065

Outcast

56

13

0.232142857

Resurrectionist

67

14

0.208955224

Ten Thunders

50

9

0.18

As we can see a lot of these faction are about 20% penetration based on their population of being in the top three. The question then becomes are any of these statistically different.

The answer is yes; note that for this because people life are not on the line I would use a relaxed inclusion criteria of .1 instead of the standard .05. The a logistic regression of faction predicting if you are in the top 3 has a P value of .08. Both Neverborn and Gremlins when compared to Ten Thunders are different with a P value of .0078 and .0374 respectively. This simple model of faction predicting if you are in the top 3 accounts for .62% of the variation in the data.

Though I am not a fan of making use of P values as I find they over simplify this all the point estimates and confidence intervals are a little crazy. The Odds Ratio Point Estimate of such as for Neverborn Vs Ten Thunders is between from .99 to 10.07 for it 95% confidence interval. What does this mean, Ten Thunders are somewhere between .99 times to 10 times more likely of getting into the top 3 when compared to Neverborn.

Addressing bias real quick. First the fact that I used .1 and not .05 as an inclusion criteria, this can have a looser degree of scrutiny as the negatives of being wrong are slight. Next is my use of predicting in the top 3, I did look at models that use 75 point or higher and models that did the top quartile which was actually about 73.4. In the end I decided top 3 would be less bias or subject to personal influence as most people I know can agree that top 3 is the placing goal in a tournament. I found months also did not matter in various models.

Further area of study. I would love to get data on the Masters used, the faction they fought, the strategy, and the points scored or differential from each round. Later looking at this by years and see if that causes a difference could be interesting.

I might be looking at UK data latter but getting it usable is a pain as I have to manually fill in the factions.

If you have any question I will put them in the bottom.

  • Like 7
Link to comment
Share on other sites

  • 3 weeks later...

It's very interesting however I'm curious on a longer trend to see if the names show up repeatedly, but across multiple factions which skew the result. Also a player may travel to multiple events with his single faction, thus skewing the results. 

As an example, and without looking at the data, I'd guess Alexander Schmidt is quite likely responsible for many of those gremlin showings. Something like Neverborn or Guild has very striking models and both are often recommended as a solid first faction for new players all things being equal.

i get that you did lead with the hypothesis that better players would gravitate toward stronger factions, but many of them have picked up their faction as a result of not wanting to have the same faction as their group.

anyway really interesting analysis, thanks for sharing!

Link to comment
Share on other sites

You got my hypothesis wrong.

Your think it is:

25 minutes ago, Somnicide said:

i get that you did lead with the hypothesis that better players would gravitate toward stronger factions, but many of them have picked up their faction as a result of not wanting to have the same faction as their group.

But it is stated as:

On 12/13/2016 at 11:28 AM, D_acolyte said:

 The theory is what do people tend to think will do in tournaments vs what actually is doing well in the tournaments.

I tend to associate this with level of mastery or confer with a faction not how strong a faction is.

27 minutes ago, Somnicide said:

It's very interesting however I'm curious on a longer trend to see if the names show up repeatedly, but across multiple factions which skew the result. Also a player may travel to multiple events with his single faction, thus skewing the results. 

As an example, and without looking at the data, I'd guess Alexander Schmidt is quite likely responsible for many of those gremlin showings. Something like Neverborn or Guild has very striking models and both are often recommended as a solid first faction for new players all things being equal.

The issue with braking it down by name is that you are stratifying your data and will probably have very few observation for most name this will most likely lead to meaningless results. I did think of making a skill level variable but there is just not enough data for the USA to do so, it would also be very subjective and probably wrong in a lot of cases when "new" players like myself who have been playing from gen 1 on suddenly show up for the first time in a tournament on the site.

If you feel that it is important then by all means go and make a data set to look at it.

Link to comment
Share on other sites

  • 1 month later...
On 1/2/2017 at 8:11 AM, D_acolyte said:

w observation for most name this will most likely lead to meaningless results. I did think of making a skill level variable but there is just not enough data for the USA to do so, it would also be very subjective and probably wrong in a lot of cases when "new" players like myself who have been playing from gen 1 on suddenly show up for the first time in a tournament on the site.

If you feel that it is important then by all means go and make a data set to look at it.

I wanted to add to/discuss this critique. 

You say at the beginning that " My father and I got into a conversation about which faction are the strongest in a tournament. "

You then say "  The theory is what do people tend to think will do in tournaments vs what actually is doing well in the tournaments. " I think from this that you mean your theory is "There is a discrepancy between the perception of what factions have performed well historically and what factions have actually performed well historically."

You can test this theory on the data set. 

I think, intuitively people see something missing without player involved, but may not immediately be able to figure out their objection. I thought through it for me, as that was my initial reaction and got to this:  the point of bringing in person to the discussion is that, because (arguable) players have far more of an effect on placement than faction, the theory can be tested historically but has no predictive value.

i think that's the response to people saying "but what about player". Of course player matters (MORE), and to have any kind of predictive validity the model would need to include faction and player. The model is not predictive, it's testing one historical theory about the disconnect between perception of history and history and the findings support or undermine that specific theory. It doesn't say anythign about which faction is "best" or most competitive.

 

Link to comment
Share on other sites

Player skill obviously matters a lot. What would be interesting though would be a statistical analysis that looks at whether any factions have better win rates while also controlling for player skill. That's the key the is doing to regressions while accounting for (and thereby eliminating player skill as a variable to try to get at the underlying faction variable).

Link to comment
Share on other sites

14 hours ago, Kolath said:

Player skill obviously matters a lot. What would be interesting though would be a statistical analysis that looks at whether any factions have better win rates while also controlling for player skill. That's the key the is doing to regressions while accounting for (and thereby eliminating player skill as a variable to try to get at the underlying faction variable).

Player skills seems like a tricky variable to isolate. You can use rankings or placings in the tournament to get something similar to it but those stats are also depebdent on other variables and don't represent pure skill.

  • Like 1
Link to comment
Share on other sites

On ‎2‎/‎22‎/‎2017 at 7:01 PM, Romes said:

I wanted to add to/discuss this critique. 

You say at the beginning that " My father and I got into a conversation about which faction are the strongest in a tournament. "

You then say "  The theory is what do people tend to think will do in tournaments vs what actually is doing well in the tournaments. " I think from this that you mean your theory is "There is a discrepancy between the perception of what factions have performed well historically and what factions have actually performed well historically."

No, I actually means what I said. People have a theory about what perform well in tournaments. With what limited data there is I can pit that vs what the data indicates and tell you the probability of it happening. I am not using the word historically because for various reasons. The US data can not be used to make a multi year trend nor examine more then this limited snap shot in time. As this snap shot at the time of the original post was the GG 2016 life time which was still going on it is not historic but concurrent. Now if I went back and looked at GG 2016 vs GG 2017 then I would describe the GG 2016 data as historic.
 

 

Link to comment
Share on other sites

6 hours ago, Ludvig said:

Player skills seems like a tricky variable to isolate. You can use rankings or placings in the tournament to get something similar to it but those stats are also depebdent on other variables and don't represent pure skill.

I am going to chime in because I can.

I feel that people either over simplify player skill or are often trying to talk about familiarity with the game. Player skill is a latent variable or a variable not directly observed but are rather inferred.  This means it can be very subjective. I will now brake down my opinion on how to model this.

Let me define some terms and put my shorthand for it. Note this is very subjective.

First is familiarity with the games as a whole: simply number of games played sort of works. F(MAL2end)

Other miniature games played: Warhammer, 40k, warmachine, guild ball and others can teach a lot about how to play, read rules and resource management. MG

Strategy computer and board games played: It like the other miniature games played, in fact sometimes even more so. Ex: starcraft 1 & 2, battle realms, total war (rome and forward), Warcraft 1-3, C&C 1-4,  Homeworld, and many more. You can also brake this one down into 2 groups. SG

Luck: number of times they get extremely normal cards. Mine tends to be bipolar. Luck

Historical knowledge: This may sound odd but I have found knowledge of warfare in history to be helpful in formulating my plans. HK

I think that does it for overall components….. But what about weights, can a player have a different skill level with other factions or even other masters? Well yes, in fact we can go all the way down to the granularity with per model basis. Could they have a different skill level in a different Meta? Well yes. I would probably depict these as weights to the entire equation. We can probably have these go from .5 to 2 to represent familiarity or just other knowledge effect even if you do not play them, are not in that meta, or do not play that faction. For sake of easy I will call these weights category or Cat fallowed by faction, meta or master. Where to put theses weights is debatable.

player skill=( F(MAL2end) *Cat_faction_played * Cat_master_played + MG+ SG+ Luck+ HK) * Cat_meta_played

Even this model is incorrect, it sort of goes with it being a latent variable. This is also a subjective model with factors I feel are important. I would also love to try and make a data set that calculates someone’s player skill but it is not probably. Also how little a difference is there to actually be a significant difference is another important thing I would like to know, answer is probably impossible to know.

Even if you get totally new players to the game they will have a difference in skill.

Link to comment
Share on other sites

6 hours ago, ttsgosadow said:

Interesting read!

Looking forward to the UK data :)

The NL scene isnt big enough to do any sensical (is that a word?) analysis.

Not sure when or if I will get to the UK data. Partly because it has little effect on me.

Though I might try to do a 2016 vs 2017 USA one in December, I might add the 2016 and 2017 UK to that. I am interested in the differences between the two groups as well as those from the GG 16 vs 17.

Link to comment
Share on other sites

@D_acolyte I don't think you need to worry about being on the overly simplistic side. :D I really enjoy your breakdowns of stuff and I agree with your take on player skill, especially on the point about it being interesting to find out how small differences give large impacts on results. For example wheighting "player skill" vs "master power level" in regards to results. I won't even pretend to get into the debate on exactly what constitutes player skill but I have noticed that a few of the locals who jumped over from Magic the gathering have become brutal to face really fast. They seem to have a knack for finding exploits and aren't the least bit ashamed about powergaming. ;) 

Link to comment
Share on other sites

Thank you, I am taking a break from my large breakdown for a while as I was running out of ideas, at least the structured ones. Taking what is unstructured and reorganizing it as well as breathing a bit. Putting my words to text is not natural or easy for me, luckily I come off overconfident enough that most people I know who read it do not pick up on how nerves I am when I do them.

The one magic player that came over in my group had some real problems getting in the grove that you do not loose early game and suffered from self defeating acts but has gotten over it. Magic, at least when I played it over 15 years ago, had a lot to do about adapting and playing power moves. Often this leads to victory or defeat a lot fast then other games in terms of time not turns. A magic player can be an absolute monster if they for alpha strikes or an early mid game power. Also most magic players I know are all about that powergaming, it is the norm for there scene, which makes it so much fun to murder them in an RPG.

  • Like 1
Link to comment
Share on other sites

D_acolyte, I think your model makes a lot of sense, but unfortunately includes a lot of data about individual players that are probably not accessible (previous history, level of familiarity with a master, etc.). You are correct that player skill is a latent variable. So perhaps the better phrasing is to account for the win rate of the player. Win rate depends on a lot of variables, but I think given enough iterations we should see that some players tend to win more than others and that is the variable we'd want to control for in an evaluation of factions (or masters for that matter). Obviously a new player might be a rock star right out of the gate, but only have a handful of games reported. But given a sufficiently large sample of data, that effect should be small.

Link to comment
Share on other sites

5 minutes ago, Kolath said:

D_acolyte, I think your model makes a lot of sense, but unfortunately includes a lot of data about individual players that are probably not accessible (previous history, level of familiarity with a master, etc.). You are correct that player skill is a latent variable. So perhaps the better phrasing is to account for the win rate of the player. Win rate depends on a lot of variables, but I think given enough iterations we should see that some players tend to win more than others and that is the variable we'd want to control for in an evaluation of factions (or masters for that matter). Obviously a new player might be a rock star right out of the gate, but only have a handful of games reported. But given a sufficiently large sample of data, that effect should be small.

I think the idea of isolating player skill has to do with some masters being seen as strictly better than others as well as who you face will greatly impact how much you win so you would wheigh wins with different masters and against different players differently. Win records are of course a lot more accessible than life histories :D 

Link to comment
Share on other sites

2 hours ago, Kolath said:

D_acolyte, I think your model makes a lot of sense, but unfortunately includes a lot of data about individual players that are probably not accessible (previous history, level of familiarity with a master, etc.). You are correct that player skill is a latent variable. So perhaps the better phrasing is to account for the win rate of the player. Win rate depends on a lot of variables, but I think given enough iterations we should see that some players tend to win more than others and that is the variable we'd want to control for in an evaluation of factions (or masters for that matter). Obviously a new player might be a rock star right out of the gate, but only have a handful of games reported. But given a sufficiently large sample of data, that effect should be small.

The volume needed for that is quite a bit, especially if you want to account for all the variations in the game and it means that someone new to the system is going to assume not to have any skill even if they have been in the game for a prolong period of time. For instance I do not tend to go to large tournaments, just do not have the time, but most in my meta consider me a hard win but if I go to the a major tournament for the first time it will hit that as a new player irrelevant of my skill.

 

So I am going to ball park this: Lets assume the tournament is using GG doc for that year so we can eliminate stratagem and deployment from the unknown, which is not always the case. If we uses GIBBS sampling and Bayesian methods over frequentice we can use smaller samples and lavage our believed expectation of skills. I would still want about 30 observation per a person as a baseline, more would be better. Factoring possible things, such as a person giving in top of turn 3 (yes that has happened when I have played a person), I would up the observations by 10 at a minimum. there are about 550 people on the tournament site, so that is 550X40=2200 observations just to get what I would think is a baseline estimation of skill. That number is actually to low in all probability.

Lets say that the average tournament enthusiast goes to one every other month, that is probably high. That is 6 tournaments a year and a total of about 7 years. I always round up on things like this. That is also assuming that we do not see a major change in the game in that time.

 

If we have it broken down by round and who they played vs then we can change this timeframe by a lot, but we do not. Especially if we have the differentials from the game.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information